Question about Forensic Image Analysis

[Pangloss]

I wasn't sure where to put this, but with all the smart people we have here I thought someone might be able to send me in the right direction.

 

What I'm looking for is information in broad terms but also about something specific. I have an extensive background in desktop publishing and basic image manipulation, and a Master's degree in Computer Information Systems, so I can probably comprehend most of the technical/mathematical issues, so I'm looking for advanced/academic-level information.

 

Specifically, one of the things I'm wondering is whether or not distortive filters can be applied in reverse in order to attempt to undo damage that was intentionally done to an image in order to mask information. To use a couple of mundane examples, perhaps someone "mustaching" his or her former spouse, or a prospective underwear model covering an inconvenient tatoo. (grin)

 

A lot of these filters one sees in programs like Photoshop don't "erase" information per se, but rather cover it up with other information (another color), applied in either a random or a non-random pattern. What I'm looking for here is whether or not that pattern can be mathematically reversed, if you will, in order to have an inkling of what the old information used to be.

 

Since the algorithm for applying that blockage is known, it occurs to me that, given a sufficiently large data sample (a lot of pixels), you might be able to glean the original information, assuming that each pixel is reverse-engineered independently from the rest (i.e. not a cumulative thing). Am I way off base here in thinking this?

 

Any suggestions of literature or papers on the subject would be appreciated, as well as any other kind of feedback. Thanks!

[In My Memory]

Pangloss,

 

I know for a fact you can distort and reverse distortion with something really simple, like changing an images hue, but I dont think its possible to reverse engineer something more complicated like smearing a image or even reversing a blur.

 

Probably, most distortions are one-way only because they loose pixels and original image data from compressing the image. Its just like if you resize a picture smaller, you'll lose pixels and never be able to get them back, you cant make the image bigger and expect resolution to magically fill in all the blank spots. You cant undo shrinking a picture because the information doesnt exist in the image anymore.

 

Almost any distortion like adding a "fisheye lens" or a swirling effect is going to compress a part of your image, so you're going to lose pixels no matter what, and you'll lose the original image data. When you lose the pixels, you can never undo a distortion.

 

You also couldnt reverse an image that was distorted in a non-mathematical way. For instance, using a paintbrush tool to change the colors of the pixels doesnt distort the image mathematically, it distorts the image by changing the pixel colors directly. You might be able to reverse this kind of distortion if the new color was semi-transparent, but not reverse it entirely, for instance if the color (255, 100, 0) was darkened to yield the new color (155, 0, 0) then you cant get back the original G value (if you brighten the image again you'll get a completely new color). If the pixels are shaded and one or more of their RGB values goes to 255 or 0, then you cant get it back.

 

If a distortion has to initialize some kind of random seed each time it executes, then you probably couldnt undo the distortion without knowing both the seed it used and the algorhythm. And if the distortion depends on the users input, such as dragging the mouse cursor across the image, then you have to know the exact X-Y coordinates of the mouse cursor (and probably the speed the cursor was dragged) to undo the distortion. This only makes the task or reversing a distortion 1000x more complicated, if its even possible at all.

[Pangloss]

That's a good description of the basic problem set. But I can't help but wonder if there may be some factors here that have gone more or less ignored because the problem seems insurmountable. It may be just a situational kind of thing, but it seems to me that repairs might be possible given the right set of data and the right conditions.

 

Just to give a brief (and highly hypothetical) example of this, let's say you had a document made under known operating conditions, say with X typewriter and Y paper. You might even have a dictionary of words known to be used in that document, and could construct a database of those words along with a set of data points describing those words mathematically (ala fingerprint "points" or DNA "markers").

 

It seems to me that under those conditions, one might apply an algorithm to the data, take a look at the results, and compare them with databased value sets.

 

Some Bayesian reasoning might be particularly applicable here.

 

Thoughts? Am I just completely reaching here, or is there a nugget of potential? And if there's a nugget, is it *only* situational?

 

Thanks for any feedback! Always fun to bounce ideas of intelligent folks.

[Pangloss]

Just to explore this a bit further, you can often guess what kind of filter or mask was applied to an image just by looking at it. This can lead you in certain directions. For example a smear or smudge filter will only use the colors already in existence in the image at that time -- no other colors need to be applied in any reversing attempts.

 

(Every filter has known parameters and values, but of course some filters would be more predictable in their outcomes than others, and one could, of course, be mistaken in guessing which filter was applied.)

 

Smudges and smears might be really good testbed examples for research along these lines, because their parameters are so few and the effects are so simple and obvious. The original data is still there, it's just been moved according to a known algorithm, with known parameters but slightly random results. But maybe this example is way too simplistic for any kind of real-world application.

[In My Memory]

Heres something you can experiment with, this is a very simple blur I wrote with MS Excel's VB:

Before blur:
----------------------------------------
|4	|55	|100	|136	|192	|
|242	|61	|56	|138	|246	|
|77	|205	|167	|6	|82	|
|----------------------------------------
|182	|143	|205	|38	|135	|
|213	|159	|106	|152	|87	|
|130	|117	|27	|177	|229	|
|----------------------------------------
|114	|89	|1	|248	|178	|
|144	|200	|229	|1	|11	|
|148	|178	|245	|165	|54	|
|----------------------------------------
|9	|174	|239	|110	|9	|
|195	|50	|91	|164	|130	|
|104	|206	|158	|196	|177	|
|----------------------------------------
|250	|172	|182	|194	|62	|
|138	|206	|25	|118	|90	|
|203	|5	|66	|46	|86	|
|----------------------------------------



After blur:
----------------------------------------
|43	|77	|75	|90	|56	|
|75	|93	|75	|87	|69	|
|59	|80	|78	|76	|55	|
|----------------------------------------
|65	|99	|113	|137	|103	|
|113	|157	|122	|114	|71	|
|95	|144	|143	|128	|79	|
|----------------------------------------
|79	|128	|139	|129	|80	|
|107	|154	|128	|108	|61	|
|98	|146	|163	|159	|111	|
|----------------------------------------
|90	|137	|157	|136	|89	|
|104	|142	|120	|95	|57	|
|94	|146	|141	|133	|80	|
|----------------------------------------
|67	|114	|119	|88	|42	|
|65	|78	|73	|69	|56	|
|58	|82	|75	|81	|56	|
|----------------------------------------

The blur works like this: each new RGB value is the average of all the RGB values of all its nearest surrounding pixels (if its missing a pixel on either side, those values are ignored, or strictly speaking they are considered black).

 

For instance, the pixel in center is calculated like this:

R = (143 + 205 + 38 + 89 + 1 + 248 + 174 + 239 + 110) / 9 = 139

G = (159 + 106 + 152 + 200 + 229 + 1 + 50 + 91 + 164) / 9 = 128

B = (117 + 27 + 177 + 178 + 245 + 165 + 206 + 158 + 196) / 9 = 163

And the pixel on the bottom-right corner is calculated like this:

R = (110 + 9 + 194 + 62) / 9 = 42

G = (164 + 130 + 118 + 90) / 9 = 56

B = (196 + 117 + 46 + 86) / 9 = 56

 

I dont know if its possible to undo the kind of very simple blur above, I've tried figuring it out and I just cant. There are up to 9 unknowns for every pixel, and the color of each pixel depends on its surrounding neighbors (whose colors you dont know either). While there is probably only 1 original configuration that can account for the distorted configuration, I dont have any idea how to start with the distortion and work backwards to the original.

 

Try writing your own distortions and seeing if you can undo them

[Cap'n Refsmmat]

With this sort of topic, it all depends.

 

As IMM has showed, there are certain things that would be incredibly difficult to reverse. There's also the problem of people just using the Paintbrush tool to paint over things, in which case that area is changed to color x entirely and probably isn't recoverable.

 

I suppose the only way to find out would be to dissect Photoshop or the GIMP and see if you can work out a way to undo some of the effects. It's apparent you can't get all of them, but you could probably find a few recoverable ones.

 

edit: heh, this thread seems to be dominated by moderators...

[Pangloss]

I appreciate the feedback. The blur table you made is great, IMM, thank you. I'll bet that was fun to do.

 

There's one aspect of this that I probably haven't explained very well. I'm not actually looking for an exact, 1:1 reversal of the effects. What I'm pondering here is the possibility that a heavy amount of processing and a lot of assumptions and situational data might allow us to bash the image around enough that a human eye might recognize a sufficient amount of the original image to make a guess about what it used to contain.

 

Investigators sometimes clean up digital imagery, say from an ATM camera, and use it to try to inform the public about criminals on the prowl. Similarly, kidnapped children's photos are sometimes "age enhanced" to show what they might look like today. What I'm looking for is more along those lines. Might it be possible to shake and bake the data enough that a clue about the original data might be gleaned that might otherwise have been missed?

 

I wonder if maybe I need a tangential exploration. Something like facial reconstruction, which is an interesting subject in itself, or perhaps diving into how hackers prosecute encrypted data attacks, might be of some use here (but hackers are looking for exact 1:1 matches so that may be of limited use to me).

[Pangloss]

Also, wouldn't interpolation be of use here? This occurs to me because of something IMM mentioned earlier, which is that when you shrink a file you throw data away, which of course is certainly the case. But when I read that the first thing that leaped into my mind is that Photoshop has a handy, built-in interpolative function that allows you to make an image larger.

 

And in many cases, the resulting image looks just fine, because either the image data was complimentary to the interpolative formula, or because the amount of data added was insufficient to trigger a human-noticable difference in the image.

 

Of course a simple interpolative approach won't reconstruct missing information when there is no hint of what the missing information might have been. It's great for doubling known pixel values, but no good at all for creating complex data of previously unknown values.

 

But what I'm actually suggesting here is that there would be known (or assumed) values -- information known to the analyst but not available in the picture, such as an eyewitness statement about hair or eye color, or the time of day, or the make/model/year of an automobile. What I'm pondering here are tools to plug that kind of information into an interpolative/extrapolative technique and produce a result that a human being might be able to analyze in a new and informative way.

[Dak]

I dont know if its possible to undo the kind of very simple blur above, I've tried figuring it out and I just cant. There are up to 9 unknowns for every pixel, and the color of each pixel depends on its surrounding neighbors (whose colors you dont know either). While there is probably only 1 original configuration that can account for the distorted configuration, I dont have any idea how to start with the distortion and work backwards to the original.

 

|a	|b	|c	|d	|e	|
|----------------------------------------
|f	|g	|h	|i	|j	|
|----------------------------------------
|k	|l	|m	|n	|o	|
|----------------------------------------
|p	|q	|r	|s	|t	|
|----------------------------------------
|u	|v	|w	|x	|y	|

 

 

will blur into:

 

|A	|B	|C	|D	|E	|
|----------------------------------------
|F	|G	|H	|I	|J	|
|----------------------------------------
|K	|L	|M	|N	|O	|
|----------------------------------------
|P	|Q	|R	|S	|T	|
|----------------------------------------
|U	|V	|W	|X	|Y	|

 

if we say that # = the number of squares surrounding the square + 1 (for the square itself), then

 

A = (a+b+f+g)/#

 

B = (a+b+c+f+g+h)/#

 

or

 

A# = a+b+f+g

 

B# = a+b+c+f+g+h

 

rearranging gives

 

A# = a+b+f+g

B# = a+b+f+g+c+h

 

combining gives

 

A#-B# = c+h

 

as A, B, and # are known, the sum of c+h can be worked out.

 

similarly, the difference between A# and F# would be equivelent to k+l,

 

Now, if we look at G.

 

G# = a+b+c+f+g+h+k+l+m

 

rearranging gives

 

G#-(c+h)-(k+l) = a+b+f+g+m

 

G and # are known from the blurred picture; (c+h) and (k+l) were previously worked out. and, a+b+f+g = A#, so

 

G#-(c+h)-(k+l)-A# = m

 

if you carry on like that, it'll probably be possible to deduce every original value from the blurred picture.