Is it possible to search a file based on the content?

[fredreload]

For instance, I want to search a movie based on a few scenes, or I want to search for a music based on a few tones, all using binaries

 

 

P.S. This would be tedious, or does it take too long?

Edited October 31, 2016 by fredreload

[Strange]

It depends what you want to search for. If you have an image from the video, then it should be relatively simple to find the most closely matching image from the video stream.

 

If you want to do something like "show me all video clips with dogs in" then the video content would have to be analysed and indexed in some way. This might be manual (by the creator of the video) or using some sort of image recognition (AI) that can detect that it is a scene with a man and a dog so when you search for "dog" it shows you that scene. I'm sure Google are working on this - if they haven't already done it.

[fredreload]

It depends what you want to search for. If you have an image from the video, then it should be relatively simple to find the most closely matching image from the video stream.

 

If you want to do something like "show me all video clips with dogs in" then the video content would have to be analysed and indexed in some way. This might be manual (by the creator of the video) or using some sort of image recognition (AI) that can detect that it is a scene with a man and a dog so when you search for "dog" it shows you that scene. I'm sure Google are working on this - if they haven't already done it.

Doctor Strange, I get your point, but I am searching for the image in terms of binaries, so I would first convert the image into a binary string, then attempt to match that binaries with the binaries of the video. This way I won't have to browse through the video to find the dog clip I want. Unfortunately, I am not sure if the binary of the image corresponds to the binary of the movie. Well, but from the most basic point of view, I would assume the binary of the dog image would resemble that of a dog drawn with 0's and 1's or I would simply translate all sorts of dog images to binaries and match that with a movie, assuming we get them in the same format. Now with this in mind, why hasn't it been done yet? Well, maybe this binary search method just takes too long or no one really cares about finding a dog in the video. I can't really say

Edited October 31, 2016 by fredreload

[fiveworlds]

That is all well and good but I would be more concerned about the potential censorship implications. If a country can analyse audio/video streams then they can search for use of bad words, nudity, profanity, and sedition.

[fredreload]

That is all well and good but I would be more concerned about the potential censorship implications. If a country can analyse audio/video streams then they can search for use of bad words, nudity, profanity, and sedition.

Well I'm pretty sure they are censored in the first place when found, reasonably, I'm pretty sure most are allowed other than those censored by the government for purposes

[fiveworlds]

Well I'm pretty sure they are censored in the first place when found, reasonably,

 

 

They are human tasks somebody watches the video then censors it. Therefore live video isn't censored however your way the government could automatically censor live streams.

[fredreload]

 

 

They are human tasks somebody watches the video then censors it. Therefore live video isn't censored however your way the government could automatically censor live streams.

Like you Fiveworld I like to watch live stream video, but that doesn't mean the live streams will get censored

[Strange]

Doctor Strange, I get your point, but I am searching for the image in terms of binaries, so I would first convert the image into a binary string, then attempt to match that binaries with the binaries of the video.

 

 

As all files, images, etc are ultimately binary patterns, every search will be "in terms of binaries" (whatever that means).

 

This way I won't have to browse through the video to find the dog clip I want. Unfortunately, I am not sure if the binary of the image corresponds to the binary of the movie.

 

It probably won't, even if you have a frame from the movie because the compression methods use mean that only a few frames are stored uncompressed. Most are stored as differences from previous frames.

 

Now with this in mind, why hasn't it been done yet?

Because it requires advanced image recognition and AI.

 

 

Mouseover text: In the 60s, Marvin Minsky assigned a couple of undergrads to spend the summer programming a computer to use a camera to identify objects in a scene. He figured they'd have the problem solved by the end of the summer. Half a century later, we're still working on it.

 

That is all well and good but I would be more concerned about the potential censorship implications. If a country can analyse audio/video streams then they can search for use of bad words, nudity, profanity, and sedition.

This is already done in law enforcement applications.

Edited October 31, 2016 by Strange

[fredreload]

I did get a chance to play around with some image recognition techniques, one being the local binary patterns. You can also train the image feature with a bag of words concept, I particularly like local binary patterns because you can just compare the histograms, a good program I've used would be in the link here. Unfortunately this only applies for gray image but not RGB image, and I did not have enough time to play with filtering techniques such as gabor filter, but the result I found using the above link did provide some really good results as the distance gets close to 0.01 or below.

 

Now with that said, local binary patterns and binary strings are different things, but a similar image comparison can be made using this method. Now assuming you've split a 60 min video with a frame by frame results you are still able to obtain the value of an image. Now if you say you wan to look through one thousand 60 min videos to find the image of the dog using local binary patterns, well that will probably take 5 years. But even then you have something like Google Image, which searches through considerably large amount of images out there.

 

So why isn't it done, I'd like to see it being applied to Youtube and other sites. But, I think it just takes too long browsing through all the binaries, well or could there be other reasons?

 

 

P.S. On another note, local binary patterns does not apply for features, so when you shrink the image to a different size it might runs into trouble, well but I haven't fully test it

Edited November 1, 2016 by fredreload

[Strange]

You might be interested in some of these:

https://en.wikipedia.org/wiki/Visual_descriptor

https://www.quora.com/What-is-the-algorithm-used-by-Googles-reverse-image-search-i-e-search-by-image (searching images is just a special case of searching video)

http://innovation.columbia.edu/technologies/cu12269_indexing-and-searching-algorithm-for-video-content-on-consumer-video-sites

[fredreload]

You might be interested in some of these:

https://en.wikipedia.org/wiki/Visual_descriptor

https://www.quora.com/What-is-the-algorithm-used-by-Googles-reverse-image-search-i-e-search-by-image (searching images is just a special case of searching video)

http://innovation.columbia.edu/technologies/cu12269_indexing-and-searching-algorithm-for-video-content-on-consumer-video-sites

Descriptor is used in feature matching, at least for OpenCV in the matching

Search by image would be more interested in scale, that is the only thing I did not accomplish

Ya, and there's video encoding, this is pretty cool, I'll take a look, thanks