Playing Android with Firebase ML kit, Tensorflow Lite and ARCore 3
Just detecting and classifying can be done by human beings, but segmenting realtime image for those you can see is pretty difficult. In this post, I am going through executing realtime segmentation with Android CameraX and Firebase ML kit.
Second Chapter: Playing Android with Firebase ML kit, Tensorflow Lite and ARCore 2
DeepLab Segmentation with Firebase and CameraX
I am going to use the starter model of deeplab version3, provided free by Google. Cool thing about the segmentation model is that it classifies objects in the image pixel by pixel, which sums up to be able to draw segments of object in the image.
The starter model consists of 21 classes, with one of it as a non-class, or background.
The main point of the application is to apply segmentation to the realtime camera vision. In order to visualize that, I will make a display of two views, camera and segmented image, vertically positioned so that it is obvious to see the augmented vision.
activity_segmentation.xml
<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:background="@color/colorPrimary"
tools:context=".SegmentationActivity">
<LinearLayout
android:id="@+id/verticalLinearLayout"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:orientation="vertical">
<RelativeLayout
android:id="@+id/relative_layout"
android:layout_width="match_parent"
android:layout_height="0dp"
android:layout_weight="9"
app:layout_constraintBottom_toTopOf="parent"
app:layout_constraintEnd_toEndOf="parent"
app:layout_constraintStart_toStartOf="parent"
app:layout_constraintTop_toTopOf="parent">
<LinearLayout
android:layout_width="match_parent"
android:layout_height="match_parent"
android:orientation="vertical">
<TextureView
android:id="@+id/cameraTextureView"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:layout_weight="3"/>
<com.shibuiwilliam.firebase_tflite_arcore.common.SegmentationView
android:id="@+id/segmentation_view"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:layout_weight="3"/>
</LinearLayout>
</RelativeLayout>
</LinearLayout>
</androidx.constraintlayout.widget.ConstraintLayout>
The cameraTextureView , the one on the top, is a simple CameraX interface with ImageAnalysis.Analyzer to run the segmentation model. How it works is quite the same as the one I did to run the previous Firebase custom model.
The view on the bottom displays the segmented image of the top camera view. What’s it doing is pretty straightforward; just drawing images on the canvas.
SegmentationView.kt
class SegmentationView: View {
private val TAG = "SegmentationView"
var paint: Paint? = null
var canvas: Canvas? = null
var displayBitmap: Bitmap? = null
private val lock = Any()
constructor(context: Context) : super(context) {
initialize()
}
constructor(context: Context,
attrs: AttributeSet?) : super(context, attrs) {
initialize()
}
constructor(
context: Context,
attrs: AttributeSet?,
defStyle: Int) : super(context, attrs, defStyle) {
initialize()
}
private fun initialize() {
paint = Paint()
canvas = Canvas()
Log.i(TAG, "initialized Segmentation View.")
}
override fun onDraw(canvas: Canvas) {
super.onDraw(canvas)
synchronized(lock){
Log.i(TAG, "display size: ${displayBitmap!!.width}, ${displayBitmap!!.height}")
canvas.drawBitmap(displayBitmap!!, 0.0f, 0.0f, paint)
}
}
fun set(bitmap: Bitmap){
synchronized(lock){
displayBitmap = bitmap
}
postInvalidate()
}
}Segmentation
The original model can be downloaded from the url. The segmentation is executed as Firebase ML kit custom model. Just like the previous image classifier, it should be uploaded to custom model in the Firebase console.
To do the segmentation on the camera view, imageProxy , as is the custom model did, you define the input and output format and convert the input image to FirebaseModelInputs .
ImageSegmentation.kt
private fun initializeIO(){
val inputDims = intArrayOf(dimBatchSize,
dimImgSize,
dimImgSize,
dimPixelSize)
val outputDims = intArrayOf(1,
dimImgSize,
dimImgSize,
numClasses)
val dataType = FirebaseModelDataType.FLOAT32
dataOptions = FirebaseModelInputOutputOptions
.Builder()
.setInputFormat(0, dataType, inputDims)
.setOutputFormat(0, dataType, outputDims)
.build()
Log.d(TAG, "Configured input & output data for the custom image segmentation.")
}@Synchronized
private fun convertBitmapToBuffer(bitmap: Bitmap): ByteBuffer {
return ImageUtils.convertBitmapToBuffer(
bitmap,
numOfBytesPerChannel,
dimBatchSize,
dimPixelSize,
false,
127.5f,
127.5f)
}private fun generateSegmentationInputs(image: Bitmap): FirebaseModelInputs {
return FirebaseModelInputs
.Builder()
.add(convertBitmapToBuffer(image))
.build()
}
The segmentation can be executed asynchronous or use Tasks.await to wait for the result.
ImageSegmentation.kt
@Throws(FirebaseMLException::class)
internal fun segment(image: Bitmap): Task<FirebaseModelOutputs> {
if (!initialized || interpreter==null) {
initialize()
}
return interpreter!!.run(generateSegmentationInputs(image), dataOptions)
}
@Throws(FirebaseMLException::class)
internal fun segmentAwait(image: Bitmap,
awaitMilliSeconds: Long=this.awaitMilliSeconds): FirebaseModelOutputs? {
if (!initialized || interpreter==null) {
initialize()
}
try{
return Tasks.await(segment(image),
awaitMilliSeconds,
TimeUnit.MILLISECONDS)
}
catch (ex: Exception){
Log.e(TAG, "${ex.printStackTrace()}")
return null
}
}The model’s input dimension is 1*257*257*3, meaning [batch size] * [height] * [width] * [color dimension], and output dimension is 1*257*257*21, [batch size]*[height] * [width] * [class]. The last class of the output is the probability of class for each pixel. Coloring the pixel of the input image, or overlaying the image, based on the class will create the segmentation of the image.
To overlay the original image, there are several choices you can take.
- color all the classes in different colors, including background
- differentiate object with coloring background
- color only the objects and no color for background
- delete background and leave the objects without coloring
- etc
These strategies can be realized with combination of color configuration and porterduff mode of the Android view.
In order to color the image, you still have to extract the result of 1*257*257*21, [batch size]*[height] * [width] * [class], and classify pixels. Then you have to mask the original image with the classified pixels.
ImageSegmentation.kt
internal fun extractSegmentation(firebaseModelOutputs: FirebaseModelOutputs):
Array<Array<Array<FloatArray>>>{
return firebaseModelOutputs
.getOutput<Array<Array<Array<FloatArray>>>>(0)
}
internal fun postProcess(results: Array<Array<Array<FloatArray>>>): Bitmap{
val output = Bitmap.createBitmap(dimImgSize,
dimImgSize,
Bitmap.Config.ARGB_8888)
for (y in 0 until dimImgSize){
for (x in 0 until dimImgSize){
output.setPixel(x,
y,
activeColor[results[0][y][x].indexOf(results[0][y][x].max()!!)])
}
}
return output
}
internal fun maskWithSegmentation(originalBitmap: Bitmap,
maskingBitmap: Bitmap): Bitmap{
val w = originalBitmap.width
val h = originalBitmap.height
val scaledMask =
if (w!=maskingBitmap.width || h!=maskingBitmap.height)
Bitmap.createScaledBitmap(maskingBitmap,
w,
h,
true)
else maskingBitmap
val output = Bitmap.createBitmap(w, h, Bitmap.Config.ARGB_8888)
val canvas = Canvas(output)
val paint = Paint(Paint.ANTI_ALIAS_FLAG)
paint.xfermode = PorterDuffXfermode(porterDuffMode)
canvas.drawBitmap(originalBitmap,0.0f,0.0f,null)
canvas.drawBitmap(scaledMask, 0.0f, 0.0f, paint)
paint.xfermode = null
paint.style = Paint.Style.STROKE
scaledMask.recycle()
return output
}These are some examples of visualizing the segmentation.
hmm… is that it?
It is fun applying segmentation to the realtime image, though there is an advanced thing you can do with segmentation. If you can classify pixel by pixel of the image, though it is still rough, you can crop the image without background.
For the next and last post of this series, I would like to, finally, use the Augmented Reality with the segmentation to place a cat view by cropping the real cat.
Playing Android with Firebase ML kit, Tensorflow Lite and ARCore 4
You can find the whole code of this AI and AR toy in the repository.