Disclaimer and Conditions Regarding Use of SWGDE Documents
SWGDE documents are developed by a consensus process that involves the best efforts of relevant subject matter experts, organizations, and input from other stakeholders to publish suggested best practices, practical guidance, technical positions, and educational information in the discipline of digital and multimedia forensics and related fields. No warranty or other representation as to SWGDE work product is made or intended.
As a condition to the use of this document (and the information contained herein) in any judicial, administrative, legislative, or other adjudicatory proceeding in the United States or elsewhere, the SWGDE requests notification by e-mail before or contemporaneous to the introduction of this document, or any portion thereof, as a marked exhibit offered for or moved into evidence in such proceeding. The notification should include: 1) The formal name of the proceeding, including docket number or similar identifier; 2) the name and location of the body conducting the hearing or proceeding; and 3) the name, mailing address (if available) and contact information of the party offering or moving the document into evidence. Subsequent to the use of this document in the proceeding please notify SWGDE as to the outcome of the matter. Notifications should be sent to secretary@swgde.org.
From time to time, SWGDE documents may be revised, updated, or sunsetted. Readers are advised to verify on the SWGDE website (https://www.swgde.org) they are utilizing the current version of this document. Prior versions of SWGDE documents are archived and available on the SWGDE website.
Redistribution Policy
SWGDE grants permission for redistribution and use of all publicly posted documents created by SWGDE, provided that the following conditions are met:
- Redistribution of documents or parts of documents must retain this SWGDE cover page containing the Disclaimer and Conditions of Use.
- Neither the name of SWGDE nor the names of contributors may be used to endorse or promote products derived from its documents.
- Any reference or quote from a SWGDE document must include the version number (or creation date) of the document and also indicate if the document is in a draft
Requests for Modification
SWGDE encourages stakeholder participation in the preparation of documents. Suggestions for modifications are welcome and must be forwarded to the Secretary in writing at secretary@swgde.org. The following information is required as a part of any suggested modification:
- Submitter’s name
- Affiliation (agency/organization)
- Address
- Telephone number and email address
- SWGDE Document title and version number
- Change from (note document section number)
- Change to (provide suggested text where appropriate; comments not including suggested text will not be considered)
- Basis for suggested modification
Intellectual Property
Unauthorized use of the SWGDE logo or documents without written permission from SWGDE is a violation of our intellectual property rights.
Individuals may not misstate and/or over represent duties and responsibilities of SWGDE work. This includes claiming oneself as a contributing member without actively participating in SWGDE meetings; claiming oneself as an officer of SWGDE without serving as such; claiming sole authorship of a document; use the SWGDE logo on any material and/or curriculum vitae.
Any mention of specific products within SWGDE documents is for informational purposes only; it does not imply a recommendation or endorsement by SWGDE.
1. Purpose
The purpose of this document is to provide forensic practitioners, legal experts, and those who require it, with a fundamental understanding of the technical aspects of image and video interpolation. A description of some standard interpolation algorithms is provided for clarification. This document also provides considerations for determining the best method to resize imagery in a forensic workflow and the introduction of imagery in legal proceedings.
2. Scope
This document addresses the use of interpolation algorithms and does not mention any specific software.
Prior to implementing forensic software, practitioners should validate the tools used. See SWGDE Recommended Guidelines for Validation Testing for more information.
This document is not intended to be used as a step-by-step guide for conducting a forensic examination or reaching a conclusion.
For the purposes of this document, the word “imagery” refers to a representation of a subject or object derived from video or still images.
3. Limitations
This document is not intended to be a training manual or provide a specific operating procedure.
This document is not inclusive of every algorithm used for image processing.
This document has excluded an in-depth discussion of how new values are derived from interpolation methods, including, but not limited to, the advanced mathematical processing within the listed algorithms.
4. Introduction
- Digital imagery comprises pixels that represent the captured scene. Each pixel has a tonal value defined in the file’s color space. In some cases, the digital imagery’s nominal resolution, or detail size, is insufficient for the required task, and the imagery needs to be enlarged. The only way to enlarge the imagery is to add more pixels, which cannot be randomly created. The process of adding pixels is accomplished through interpolation.
- Interpolation, as it is being used in this document, is defined as a method of image processing whereby one pixel, block, or frame is displayed or stored based on the differences between the previous and subsequent pixel, block, or frame of information. This resizing is often done to increase the apparent clarity of an image. [1]
- Mathematicians have employed interpolation of numeric values for centuries. Several digital imagery processing interpolation methods have been widely used since the 1980s, as documented in Jain K. Anil’s textbook “Fundamentals of Digital Image Processing.”[2]
- In addition to its use in legal proceedings, image interpolation is commonly used in medical diagnostic imaging, satellite imaging, and weather forecasting. Interpolation is a part of our everyday lives and provides us a clearer understanding of our world, even if we are not aware of it.
- Using known interpolation algorithms for resizing imagery assures the practitioner and the trier of fact that the process is repeatable regardless of the software application. These repeatable processes contribute to the reliability of the application and the reproducibility of the imagery. Additionally, using known methodologies with proper training can help mitigate the addition of artifacts or altering of the imagery’s content. While there may be additional pixels, the goal is to make the imagery easier to review and not change its interpretation.
- The use of interpolation or the method for that interpolation is not always visible to the end user nor documented. When imagery is of a different size than the screen that displays them, the viewing device will often interpolate the imagery to fit or fill the screen. This interpolation is commonly applied when using televisions, computer monitors, printers, and cell phones.
5. Common Interpolation Algorithms
5.1 Nearest neighbor
- This basic algorithm increases the pixel resolution of the imagery by leaving space for new pixels to be inserted and filling the new empty pixels with the same pixel values of neighboring pixels. Though additional pixels are added with this method, it is done so that no new visual (e.g., new averaged pixel values) information is created. [3] This type of interpolation is best thought of as a redundant or duplicative representation of data instead of introducing new values. Nearest Neighbor interpolation is best used with text and defined lines as it can expand jagged edges in non-linear edge detail.
- This interpolation method more accurately enlarges the imagery as it was captured; however, the pixelation will become more visible when increasing the size of each pixel and may be perceived as a misrepresentation of imagery’s natural features as an enlargement of the image increases. For example, the pixels may present the appearance of a square object when the object in the original scene is curved.
5.2 Bilinear
- This algorithm increases the resolution of imagery, but rather than replacing empty pixels with the same value of a neighboring pixel, bilinear uses data from 4 neighboring pixels, with pixels closest to the empty pixel having a more significant impact on the calculation. [2]
- While creating more natural imagery in appearance, this algorithm can cause blurring and create a slight halo around hard edges. Halo artifacts generally occur because edges of the imagery are not preserved during interpolation.
5.3 Bicubic
- This complex algorithm uses data from 16 neighboring pixels, with pixels closest to the empty pixel having a more significant impact on the calculation. As such, more weight is given to commonalities in the surrounding pixels. [2]
- This algorithm creates smoother tonal gradations than Nearest Neighbor or Bilinear and may reduce the artifacts presented by other resizing interpolation algorithms. This processing may have the benefit of generating imagery that is smoother while also preserving edge detail.
5.4 Differences in Interpolation
When the edges of different pixel values are aligned, tonal values may blend as the interpolation algorithms calculate values for new pixels, as demonstrated in figure 7. Additionally, the edge detail differences between the interpolation algorithms are also visible (figure 8).
5.5 Computer-assisted or adaptive algorithms
Recent advancements in technology have included computer-assisted and machine learning interpolation algorithms. These algorithms are often utilized in media production and presentations. Practitioners should be cautious when utilizing novel interpolation methods, as it can be challenging to identify what processes were applied to the imagery and replicate those steps with accuracy.
6. Considerations for Determining Proper Interpolation Usage
- Necessity of edge detail preservation
- In cases where the preservation of the exact encoded detail is necessary, Nearest Neighbor interpolation would generally be the preferred method due to edge blurring and halos often introduced when utilizing Bicubic or Bilinear interpolation. For example, utilizing Nearest Neighbor when obtaining measurements in an analysis may be more appropriate as the edges are unchanged.
- Conversely, if the purpose of the interpolation is to give a more accurate representation of the edge detail of the imagery, utilizing Bicubic or Bilinear may be more effective than Nearest Neighbor. For example, while Nearest Neighbor will show that a vehicle tire is only a few pixels tall, Bicubic may better show the realistic shape of that tire.
- Size of the original object being analyzed
- Utilizing Nearest Neighbor interpolation to enlarge imagery that contains small objects within (e.g., weapon in hands) may help mitigate the distortion of the object’s shape, length, or size represented by a few pixels. The Nearest Neighbor algorithm can reduce the potential of providing a misleading representation of the level of pixel detail in the original imagery.
- When applying interpolation to objects represented by more than a few pixels, Bilinear or Bicubic interpolation may better maintain smooth edges and preserve the natural appearance of the original object.
- Increments of change
- When using any type of interpolation, it is important to be cognizant of the amount of enlargement or reduction utilized. The algorithms are designed to work with whole numbers; therefore, resizing to odd increments may remove or split the original pixel within the imagery. Due to this possibility, practitioners should attempt to make even increments (e.g., 200%, 50%). This will help ensure that the algorithms are not accounting for partial pixel values in their calculations.
- The practitioner should be aware of the weight given to a single pixel when a factor greater than 4 is used to enlarge the imagery. For example, when resizing by a factor of 10, information in a single pixel may be interpolated into 100 pixels. [2]
- Introduction of new pixel values not present in the original imagery
- When utilizing any algorithm for interpolation, regardless of whether it is for enlargement or reduction, there will inherently be alterations to the number of pixels present. These alterations are accomplished by either adding new pixels or removing pixels to achieve the given task.
- When enlarging, the newly formed pixels are not randomly generated but generated based on the surrounding values of the previously recorded pixels. Depending on the method of interpolation, there may be graduation from one value to the other to achieve this (e.g., a gray pixel inserted as a transition between the original black and white pixels). However, the newly generated pixels will be a combination of the tonal values of the surrounding pixels. The imagery will not contain data that was not derived from the originals. This may be most noticeable at the edges of contrast points for the relationship at those edges to remain consistent. Depending on the algorithm utilized, this may appear as a blur or halo.
- When downsizing or removing pixels, typically through Bilinear or Bicubic, the algorithm maintains the relationship of the original values while reducing the overall number of pixels. While not typical in image or video enhancement, this reduction may be necessary for file sharing or presentation purposes (e.g., imagery is encoded larger than the native resolution of the display it will be presented on).
- A common misconception with imagery enlargement is that enlargements will increase detail [2]. While enlargement of an object may increase the visual acuity, or one’s ability to perceive subtle details within the imagery, the enlargement process does not introduce new detail absent from the original recording.
7. Considerations for legal proceedings
While interpolation may occur during the image processing and when displaying the image, it can generally be classified as happening either prior to encoding or during the decoding process. It is important to be aware of this potential to help mitigate any consequences of unintended interpolation being applied to imagery.
7.1 Interpolation During Creation of Demonstrative Imagery
- Interpolation is most controllable when applied during imagery generation. This interpolation can be applied by a third-party utility (e.g., editing software), forensic analysis tools, or in the packaging of a file by an encoder.
- Some examples where interpolation may occur in this phase include:
- Resizing/scaling imagery for analysis or presentation.
- Resizing/scaling imagery shared digitally (e.g., text, email). This may occur without notification by the sharing platform.
- Imagery restoration or enhancement (e.g., aspect ratio adjustment, lens distortion correction).
- Exporting from a proprietary video player into a standard format.
7.2 Interpolation During Imagery Presentation
- While practitioners can understand interpolation occurring during demonstrative imagery creation, several factors may be applying additional interpolation beyond the practitioner’s control. These types of interpolation happen across most applications and devices, from displaying an image on a television or computer screen to playing a video at full screen or in a minimized view.
- Considerations should be made towards the final presentation display device during the encoding process. The material in court should be presented with caution to ensure that any demonstrative imagery brought to court accurately represents the evidence as analyzed since the practitioner may not be able to control the display technology in the courtroom. Preparing exhibits with a known magnification beforehand can help eliminate the need for dynamic magnification during testimony. This will assist the trier of fact by allowing them to review imagery during deliberations consistent with the evidence presented.
- If the court’s display monitors are incapable of presenting the resolution of the original imagery, it would be recommended that the practitioner downsize the imagery to meet the display capabilities prior to presentation to prevent unknown interpolation processes from being applied by the display. Conversely, if the resolution of the display monitor exceeds that of the displayed imagery, the practitioner should determine if adjusting the resolution of the display monitor, or enlarging the source imagery using a known interpolation method, would most accurately present the evidence.
- The practitioner should be prepared to provide and speak to the original imagery as well as the interpolated demonstrative imagery by having the before and after materials available
- Some additional examples of when interpolation may occur during display include:
- Aspect ratio incompatibilities between the displayed imagery and monitor.
- Display monitor resolution settings.
- GPU Acceleration.
- Software playback application settings (e.g., texture interpolation, smooth playback, player adjustments, resizing the application).
See SWGDE Practical Considerations for Submission and Presentation of Multimedia Evidence in Court.
8. Cases in the Media
- During cross-examination of the defendant, in the State of WI v. Kyle H. Rittenhouse (2021), the prosecution attempted to display imagery using a “pinch-to-zoom” feature on an iPad. The defense objected, and the court sustained, that the method utilized had not been explained to the jury. The displayed imagery would only be admissible if the prosecution could explain the interpolation methodology. Additional questions were raised about the method of interpolating imagery submitted and the playback of imagery on a standard high-definition display versus an ultra high-definition display.
- In the trial of Rodd v Raritan Radiologic Associates, P.A., et al. 860 A.2d 1003 (2004), the use of enhanced medical films was not permitted. The court had initially permitted enlarged copies of medical films; however, it was later found that the magnification applied to the films distorted the image content. The experts on both parties agreed that they had difficulty interpreting the imagery, primarily because of the large amount of magnification. In addition, content that appeared sharp in the original was blurred in the enlargement.
9. Resources
1. Scientific Working Group on Digital Evidence. (2016, June 23). SWGDE Digital and Multimedia Evidence Glossary. SWGDE.
2. Anil K. Jain, Fundamentals of Digital Image Processing, Prentice-Hall, 1989. ISBN 0-13-336165-9
3. Ledesma, Spencer “A Proposed Framework For Forensic Image Enhancement,” University of Colorado, 2015
4. Scientific Working Group on Digital Evidence. (2020, September 17). 2020-09-17 SWGDE Practical Considerations for Submission and Presentation of Multimedia Evidence in Court_v1.0.pdf. SWGDE.
5. Scientific Working Group on Digital Evidence. (2014, September 5). SWGDE Recommended Guidelines for Validation Testing. SWGDE.
6. Heinzman, Andrew “What is ‘Upscaling’ on a TV, and How Does it Work?” 2019-07-05 https://www.howtogeek.com/427091/what-is-upscaling-on-a-tv/
History
| Revision | Issue Date | History |
|---|---|---|
|
1.0 DRAFT |
1/13/2022 |
Initial draft created and voted by SWGDE for
release as a draft for public comment. |
|
1.0 |
6/8/2022 |
Addressed comments from other committees and
clarity of some points. |
|
1.1 |
9/22/2022 |
Updated document to address comments and
proposed for release as a final publication. |
Version: 1.1 (September 22, 2022)