Two-layer Lossless HDR Coding considering Histogram Sparseness with Backward Compatibility to JPEG
An efficient two-layer coding method using the histogram packing technique with the backward compatibility to the legacy JPEG is proposed in this paper. The JPEG XT, which is the international standard to compress HDR images, adopts two-layer coding scheme for backward compatibility to the legacy JPEG. However, this two-layer coding structure does not give better lossless performance than the other existing single-layer coding methods for HDR images. Moreover, the JPEG XT has problems on determination of the lossless coding parameters; Finding appropriate combination of the parameter values is necessary to achieve good lossless performance. The histogram sparseness of HDR images is discussed and it is pointed out that the histogram packing technique considering the sparseness is able to improve the performance of lossless compression for HDR images and a novel two-layer coding with the histogram packing technique is proposed. The experimental results demonstrate that not only the proposed method has a better lossless compression performance than that of the JPEG XT, but also there is no need to determine image-dependent parameter values for good compression performance in spite of having the backward compatibility to the well known legacy JPEG standard.
The image compression method designed to provide coded data containing high dynamic range content is highly expected to meet the rapid growth of high dynamic range (HDR) image applications. Generally, HDR images have much greater bit depth of pixel values and much wider color gamut[1, 2, 3, 4]. These characteristic of HDR images are suitable for recording and/or archiving the highly valuable contents, such as masterpieces of art. For such a valuable content, HDR images should be losslessly encoded. In other words, they should be compressed without any loss that is generated during compression procedure.
Most of conventional image compression methods, however, could not efficiently compress HDR image due to its greater bit depth and uncommon pixel format including a floating point based pixel encoding. Several methods have been proposed for compression of HDR images[5, 6, 7, 8, 9, 10, 11, 12, 13] and ISO/IEC JTC 1/SC 29/WG 1 (JPEG) has developed an international standard referred to as JPEG XT[14, 15, 16, 17, 18] for compression of an HDR image. JPEG XT has been designed to be backward compatible with legacy JPEG with two-layer coding; a base layer for tone-mapped LDR image is compressed by the legacy JPEG encoder and an extension layer for residual data consists of the result of subtraction between a decoded base layer image and an original HDR image is compressed by the JPEG-like encoder. This backward compatibility to legacy JPEG allows legacy applications and existing toolchains to continue to operate on codestreams conforming to JPEG XT. Although this two-layer coding procedure makes it possible to compress HDR images with the backward compatibility and the extension layer contributes the improvement of the decoded image quality in lossy compression, its lossless compression performance is not better than that of the other existing methods for HDR image compression with single coding layer procedure. The ISO/IEC IS 18477-8, which is known as the JPEG XT part 8, makes it possible to encode HDR images losslessly with such a two-layer coding procedure. In this part 8, it is required to find a combination of the parameter values which gives a good lossless compression performance. The combination could be dependent on input HDR images. That is, finding the combination is required to compress HDR images losslessly and efficiently.
In Refs.[22, 23, 24, 25, 26, 27, 28, 29, 11, 30, 12], the sparseness of a histogram of an image is used for efficient compression. ‘Sparse‘ histogram means that not all the bins in a histogram are utilized. It is well known that a histogram of an HDR image shows a tendency to be sparse[11, 12]. In Ref.[12, 31, 32], two-layer lossless coding of HDR images has been proposed, however, methods in [12, 31] are not backward compatible with legacy JPEG. Ref. has described the JPEG compatible two-layer lossless coding with histogram packing, however, the encoder for the extension layer has been fixed. In addition, only the performance for HDR images with floating point pixels has been examined and the effect of the histogram packing has not been investigated.
This paper proposes a new lossless two-layer method for both integer/floating-point HDR images with histogram packing and provides the investigation of the effect of the histogram packing. Codestreams produced by the proposed method consist of two layers, i.e. base layer and extension layer, where the base layer provides low dynamic range (LDR) images mapped from HDR images by a tone mapping operator (TMO), while the extension layer has the residual information for reconstructing the original HDR images. For those residual data, any lossless image encoders that can handle over 16 bits, such as JPEG 2000 and JPEG XR, could be used. In addition, the codestreams for the base layer are compatible with legacy JPEG decoders.
Ii Problems with JPEG XT lossless coding
Because we focus the lossless coding of HDR images with backward compatibility with the legacy JPEG decoders, the coding procedure of the JPEG XT part 8 is summarized and then the problem with it is described.
The blockdiagram of the part 8 encoder is shown in Fig.1. Although the pixel values of HDR images are often represented with floating point numbers, these floating point numbers are re-interpreted as integer number with IEEE floating point representation[33, 34]. This representation is exactly invertible and makes it possible to compress HDR images losslessly.
For lossless compression of HDR images, it is required to determine the values of several parameters. The first parameter is , which controls decoded image quality of base layer. The higher gives the better quality. The second parameter is the number of bits used for refinement scan. The refinement scan is used to improve precision of DCT coefficients up to 12 bit. Thus the valid range of is from to . The third parameter is . The is the number of bits used for residual refinement scan. In lossless coding procedure, the is considered as the control factor for the amount of coded data included in the residual data of the extension layer.
To achieve good lossless compression performance, the values of the parameters, , and should be carefully determined. Figure 2 shows the result of lossless compression of an HDR image by the part 8 with to , and .
Clearly, we can see there is a certain variation in the coding performance. Note that it has been confirmed that the optimal values of the parameters which give the best performance is image-dependent.
Iii Proposed method
A method using the histogram packing technique with the two-layer coding having the backward compatibility to the legacy JPEG for base layer is described in this section.
Iii-a Histogram sparseness of residual data
HDR images often have sparse histograms due to its high dynamic range of pixel values. Moreover, the histograms of the residual data in the two-layer coding in the part 8 are also sparse after subtraction of LDR data in the base layer. In this paper, this histogram sparseness is denoted as and defined by
where denotes the histogram of a pixel value , and denotes the total number of all the elements of a set . The range of is and the greater means the sparser histogram. Figure 3 and 4 show the ‘sparseness’ of the residual data of two types of HDR images having floating-point and integer pixel values. The remarks from these figures are summarized as follows.
The sparseness depends on images and the quality factor for base layer.
The histogram of residual data tends to be sparse, especially, the value of in chroma component is higher than that in luminance.
For image signals having such a sparseness, it is well known that the histogram packing technique improves lossless compression performance[24, 25, 26, 27, 28, 29]. The main idea of the proposed method is to combine the two-layer coding structure with the histogram packing technique.
Iii-B Histogram packing
In previous subsection, it has been noted that the histograms of the residual data tend to be sparse and the reduction of the sparseness is effective to improve lossless coding performance. Figure 5 shows a histogram for Y component in the residual data of an HDR image ’BloomingGorse2.’ Horizontal axis denote the pixel values in integer number with IEEE floating point representation. After histogram packing, a histogram-packed image is obtained. In this paper, this histogram-packed image is referred to as ’index image.’ for the index image is shown in Fig. 5(a) and it is clearly considered to be dense. The index image is compressed by the lossless image encoder. The unpacking table, which is necessary to perform inverse histogram packing, is illustrated in Fig. 5(b). Obviously, it is considered as one-to-one correspondence function and monotonically increasing. That is, this table is DPCM effectiveï¼
Iii-C Encoder structure
The structure of the proposed lossless two-layer coding is illustrated in Fig.7. The coding-path to generate a base layer, which is backward compatible with the legacy JPEG, is the exactly same as the JPEG XT part 8. Note the refinement scan for the base layer is not used. Therefore, the value of is set to zero.
For the extension layer, which consists of the residual data generated by subtracting decoded base layer from the original HDR image, the coding procedure after color space conversion from RGB to YCbCr is different from the part 8 encoder. The histogram of each color component of the color converted residual data is analyzed and packed by using the histogram packing technique. Then, the packed residual data is compressed by the lossless image encoder, such as the JPEG 2000, JPEG XR etc. After the subtraction described above, the residual data for each color component could have 17 bit integers. This over 16 bit in the bit-depth is the reason for using such lossless encoders as the JPEG 2000 and the JPEG XR because they are able to accept up to 32 bit integer pixel value per component. For the inverse operation of the histogram packing, unpacking table is sent to the decoder. The unpacking table is one-to-one correspondence function between the packed index value and the original pixel value. Since this is monotonically increasing, DPCM and bzip2 compression are performed to reduce the data amount of the table.
The base layer which is compatible with the legacy JPEG, the extension layer consists of the lossless JPEG 2000 or JPEG XR codestream, and the compressed unpacking table are multiplexed into single codestream and it is sent to the decoder.
Iv Experimental results
To verify the effectiveness of the proposed method, the lossless compression performance in terms of bitrate of the generated codesrtreams was evaluated and compared with that of the JPEG XT part 8.
Images having both floating-point and integer pixel values were selected for the experiments. For floating-point images, four of images common to HDR related experiments were collected. For integer images, four of the ITE test images were used. The specifications of these test images are summarized in Table I. Although some of floating-point images have full precision float value for their pixel value, we have converted the values into half precision float because the JPEG XT encoder only accepts half precision floating point pixels as its inputs. Note that image names are all represented by the index shown in TableI. The first character of the index means the type of pixel values; “f” is for floating-point and “i” is for integer. The second number stands for each image’s name. For the JPEG XT part 8 encoder, the reference software[37, 38] available from the JPEG committee was used. For the proposed method, the modified encoder of the reference software, whose coding path for the residual data was changed to have the histogram packing and JPEG 2000/JPEG XR encoder, was used. The Kakadu software and the reference software of the JPEG XR were used as those encoders that were used to compress the histogram-packed residual data. The lossless performances of the proposed method and the JPEG XT part 8 were evaluated with several values of (quality factor of LDR image) and (number of refinement bits for base layer). For the JPEG XT, another parameter, the effect of (number of refinement bits for extension layer), was also evaluated.
|Pixel value type||Index||Name||bpp||Size|
Iv-B Results and remarks
Iv-B1 Lossless performance
Figures 8 and 9 show the results of lossless bitrate with the proposed method with the different LDR quality and those with the JPEG XT part 8 with the different parameter values of , and . From these results, it is clearly confirmed that the proposed method shows the best lossless performance regardless of images and those pixel value types, the values of . Figures 8 and 9 show the results of lossless bitrate with the proposed method with the different LDR quality and those with the JPEG XT part 8 with the different parameter values of , and . From these results, it is clearly confirmed that the results of the proposed method show the better lossless performance regardless of images and those pixel value types, the values of LDR quality . It is worth noting that the values of and/or should be carefully determined for the JPEG XT. For example, in Fig. 7(b), the bitrates of JPEG XT with the combination of the refinement parameters from to are better than those with the other combinations, however, its bitrates over are the worst. This means the best combination of and depends on the LDR quality and the input image. On the other hand, the proposed method with the JPEG 2000 encoder gives the first best performance. The second best is the result of proposed method with the JPEG XR encoder. Although there is some difference between the results, those two types of the proposed method give the lower bitrate than those obtained by the JPEG XT encoder, even though there is no dependency on the LDR and the input image. Note that it is verified that the ratios of the data amount for the unpacking table to the total bitrate are less than 0.4% at maximum.
Iv-B2 Effect of Histogram packing
To verify the effectiveness of histogram packing for lossless compression, lossless bitrates in single layer JPEG 2000 with/without histogram packing and two layer coding without histogram packing and the proposed method were examined. Figure 10 shows those lossless bitrates for floating-point HDR images. The remarks from this figure are two: a) Clearly, lossless bitrates with histogram packing are smaller than those without histogram packing. It has been confirmed that the use of JPEG XR encoder instead of JPEG 2000 provide the same trend and the results for integer HDR images denote the same tendency. b) The histogram sparseness is effective to improve lossless compression performance of HDR images having sparse histogram, regardless of whether the encoder structure is two or single layer.
A novel method using the histogram packing technique with the two-layer coding having the backward compatibility with the legacy JPEG for base layer has been proposed in this paper. The histogram packing technique has been used to improve the performance of lossless compression for HDR images that have the histogram sparseness. The experimental results in terms of lossless bitrate have demonstrated that the proposed method has a higher compression performance than that of the JPEG XT Part 8. Unlike the JPEG XT Part 8, there is no need to determine image-dependent values of the coding parameters to achieve good compression performance. Moreover, the base layer produced by the proposed method has the backward compatibility with the legacy JPEG standard, which is one of the most spread image format.
-  E. Reinhard, W. Heidrich, P. Debevec, S. Pattanaik, G. Ward, and K. Myszkowski, High dynamic range imaging - acquisition, display and image based lighting, 2nd edn. Morgan Kaufmann, Burlington, 2010.
-  A. Artusi, T. Richter, T. Ebrahimi, and R. K. Mantiuk, “High Dynamic Range Imaging Technology [Lecture Notes],” IEEE Signal Process. Mag., vol. 34, no. 5, pp. 165–172, Sept 2017.
-  A. Artusi, F. Banterle, T. O. Aydin, D. Panozzo, and O. Sorkine- Hournung, Image Content Retargeting: Maintaining Color, Tone, and Spatial Consistency. AK Peters, Ltd (CRC Press), 2016.
-  F. Banterle, A. Artusi, K. Debattista, and A. Chalmers, Advanced High Dynamic Range Imaging: Theory and Practice, 1st ed. AK Peters, Ltd (CRC Press), Feb. 2011.
-  G. Ward and M. Simmons, “JPEG-HDR: A Backwards-compatible, High Dynamic Range Extension to JPEG,” in ACM SIGGRAPH 2006 Courses, ser. SIGGRAPH ’06. New York, NY, USA: ACM, 2006. [Online]. Available: http://doi.acm.org/10.1145/1185657.1185685
-  R. Xu, S. N. Pattanaik, and C. E. Hughes, “High-dynamic-range still-image encoding in JPEG 2000,” IEEE Comput. Graph. Appl., vol. 25, no. 6, pp. 57–64, Nov 2005.
-  D. Springer and A. Kaup, “Lossy compression of floating point high-dynamic range images using JPEG2000,” in Proc.SPIE, vol. 7257, 2009, pp. 7257 – 7257 – 11. [Online]. Available: http://dx.doi.org/10.1117/12.805315
-  Y. Zhang, D. Agrafiotis, and D. R. Bull, “High Dynamic Range image amp; video compression a review,” in 2013 18th International Conference on Digital Signal Processing (DSP), July 2013, pp. 1–7.
-  I. R. Khan, “Two layer scheme for encoding of high dynamic range images,” in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing, March 2008, pp. 1169–1172.
-  T. Jinno, M. Okuda, and N. Adami, “New local tone mapping and two-layer coding for HDR images,” in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing, March 2012, pp. 765–768.
-  M. Iwahashi and H. Kiya, “Efficient lossless bit depth scalable coding for HDR images,” in APSIPA ASC, 2012 Asia-Pacific, Dec 2012, pp. 1–4.
-  M. Iwahashi and H. Kiya, “Two layer lossless coding of HDR images,” in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing, May 2013, pp. 1340–1344.
-  H. Kobayashi and H. Kiya, “An Extension of JPEG XT with JPEG2000,” in 2017 IEEE International Conference on Consumer Electronics - Taiwan (ICCE-TW), June 2017, pp. 373–374.
-  T. Richter, “On the standardization of the JPEG XT image compression,” in Picture Coding Symposium (PCS), Dec 2013, pp. 37–40.
-  A. Artusi, R. K. Mantiuk, T. Richter, P. Hanhart, P. Korshunov, M. Agostinelli, A. Ten, and T. Ebrahimi, “Overview and evaluation of the JPEG XT HDR image compression standard,” Journal of Real-Time Image Processing, Dec 2015. [Online]. Available: https://doi.org/10.1007/s11554-015-0547-x
-  A. Artusi, R. K. Mantiuk, T. Richter, P. Korshunov, P. Hanhart, T. Ebrahimi, and M. Agostinelli, “JPEG XT: A Compression Standard for HDR and WCG Images [Standards in a Nutshell],” IEEE Signal Process. Mag., vol. 33, no. 2, pp. 118–124, March 2016.
-  “Information technology — Scalable compression and coding of continuous-tone still images – Part 1: Scalable compression and coding of continuous-tone still images,” ISO/IEC 18477-1, 2015.
-  T. Richter, A. Artusi, and T. Ebrahimi, “JPEG XT: A New Family of JPEG Backward-Compatible Standards,” IEEE Multimedia, vol. 23, no. 3, pp. 80–88, July 2016.
-  “Information technology — Digital compression and coding of continuous-tone still images: Requirements and guidelines,” International Standard ISO/IEC IS-10918-1, Feb. 1994.
-  R. K. Mantiuk, T. Richter, and A. Artusi, “Fine-tuning JPEG-XT compression performance using large-scale objective quality testing,” in Proc. IEEE Int. Conf. on Image Processing, Sept 2016, pp. 2152–2156.
-  “Information technology — Scalable compression and coding of continuous-tone still images – Part 8: Lossless and near-lossless coding,” ISO/IEC 18477-8, 2016.
-  M. Iwahashi, H. Kobayashi, and H. Kiya, “Lossy compression of sparse histogram image,” in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing, March 2012, pp. 1361–1364.
-  S. Minewaki, M. Iwahashi, H. Kobayashi, T. Yoshida, and H. Kiya, “Near lossless coding of sparse histogram images based on zero-skip quantization,” Multimedia Tools and Applications, Sep 2017. [Online]. Available: https://doi.org/10.1007/s11042-017-5082-2
-  A. J. Pinho, “On the impact of histogram sparseness on some lossless image compression techniques,” in Proc. IEEE Int. Conf. on Image Processing, vol. 3, 2001, pp. 442–445 vol.3.
-  P. J. S. G. Ferreira and A. J. Pinho, “Why does histogram packing improve lossless compression rates?” IEEE Signal Process. Lett., vol. 9, no. 8, pp. 259–261, Aug 2002.
-  A. J. Pinho, “An online preprocessing technique for improving the lossless compression of images with sparse histograms,” IEEE Signal Process. Lett., vol. 9, no. 1, pp. 5–7, Jan 2002.
-  A. J. Pinho, “A comparison of methods for improving the lossless compression of images with sparse histograms,” in Proc. IEEE Int. Conf. on Image Processing, vol. 2, 2002, pp. II–673–II–676 vol.2.
-  M. Aguzzi and M. Albanesi, “A novel approach to sparse histogram image lossless compression using JPEG2000,” ELCVIA Electronic Letters on Computer Vision and Image Analysis, vol. 5, no. 4, pp. 24–46, 2006. [Online]. Available: http://elcvia.cvc.uab.es/article/view/116
-  S. Jallouli, A. Masmoudi, S. Zouari, W. Puech, and N. Masmoudi, “A Preprocessing Technique for Improving the Compression Performance of JPEG 2000 for Images with Sparse or Locally Sparse Histograms,” in 25th European Signal Processing Conference, Aug 2017, pp. 1962–1966.
-  M. Iwahashi, H. Kobayashi, and H. Kiya, “Fine rate control and high SNR coding for sparse histogram images,” in Picture Coding Symposium (PCS), May 2012, pp. 205–208.
-  T. Yoshida, M. Iwahashi, and H. Kiya, “Two-Layer Lossless Coding for High Dynamic Range Images Based on Range Compression and Adaptive Inverse Tone-Mapping,” IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, vol. E101-A, no. 1, pp. 259–266, Jan 2018.
-  O. Watanabe, H. Kobayashi, and H. Kiya, “Lossless Two-Layer Coding using Histogram Packing Technique for HDR Images,” in Proc. IEEE Int. Conf. on Circuits and Systems, 2018, to be published.
-  “IEEE Standard for Floating-Point Arithmetic,” IEEE Std 754-2008, pp. 1–70, Aug 2008.
-  J. F. Blinn, “Floating-point tricks,” IEEE Comput. Graph. Appl., vol. 17, no. 4, pp. 80–84, Jul 1997.
-  T. Richter, “On the integer coding profile of JPEG XT,” in Proc.SPIE, vol. 9217, 2014, pp. 9217 – 9217 – 19. [Online]. Available: http://dx.doi.org/10.1117/12.2060316
-  The Institute of Image Information and Television Engineers, “Ultra-high definition/wide-color-gamut standard test images,” Aug 2014.
-  “Text of CD2 ISO/IEC 18477-5 (Reference Software),” ISO/IEC JTC 1/SC 29/WG 1 N71012, Feb. 2016.
-  https://jpeg.org/jpegxt/software.html.
-  “kakadu software,” http://www.kakadusoftware.com.
-  https://jpeg.org/jpegxr/software.html.