What I made

I made a AR camera application that can detect an object via camera and display its name and related words in augmented reality.

Code here:

https://github.com/shibuiwilliam/ARWithWord

Why I made it

Many AR applications displays object or image in the space, like Google’s AR search.

Although there are a plenty of AR applications that shows visual effect via camera, I see a few that deals with natural language. I agree that these applications are brilliant, while I also believe in power of words, that people recognize the world with language and the AR is one representation of the world. …


Expanding world with AR

Augmented Reality allows us to programmatically work with 3 dimensional world using camera-equipped device. ARCore is one of the libraries for augmented reality that provides simple and powerful API’s to make a new application for Android, iOS, Unity and Unreal Engine. You may find many sample applications in the ARCore website.

Common thing you do with augmented reality application is using camera as interface to change the scene you can see. You may easily think of it to use for games, but I am expecting more possibilities to AR to attract real world.

Augmented reality technology allows us to…


ARCore with TFLite Segmentation in augmented reality

Finally I am ready to play with ARCore. ARCore is a powerful library to implement augmented reality with various platforms: Android, iOS, Unreal and Unity. It provides API’s to understand the real world with camera interface including motion tracking, light estimation and object placement.

Third Chapter: Playing Android with Firebase ML kit, Tensorflow Lite and ARCore 3

placing virtual Andy on a table

You can find plenty of cool examples in the official github and tutorials. To use ARCore, you need supported devices with SDK. Refer here for the quickstart.

Duplicating object in the camera interface

While there are many examples, I would like to combine segmentation to place a duplication object…


Just detecting and classifying can be done by human beings, but segmenting realtime image for those you can see is pretty difficult. In this post, I am going through executing realtime segmentation with Android CameraX and Firebase ML kit.

Second Chapter: Playing Android with Firebase ML kit, Tensorflow Lite and ARCore 2

DeepLab Segmentation with Firebase and CameraX

I am going to use the starter model of deeplab version3, provided free by Google. Cool thing about the segmentation model is that it classifies objects in the image pixel by pixel, which sums up to be able to draw segments of object in the image.

https://www.tensorflow.org/lite/models/segmentation/images/segmentation.gif

The starter…


Object detection and image classification with Firebase ML kit

In the previous post, I have gone through running custom image classification model on Firebase ML kit. It was just classifying the whole camera screen. In the real world, the camera takes a bunch of noises in one image. On the other hand, the object you want to focus on is only a portion of the whole image. In order to focus on the object, we use object detection.

First chapter: Playing Android with Firebase ML kit, Tensorflow Lite and ARCore 1

Firebase ML kit object detection

While there are plenty of methods to utilize object detection, Firebase ML kit provides a pretrained one ready…


CameraX, TFLite and Firebase ML kit

Today, I would like to introduce combination of Tensorflow Lite, Firebase ML kit, ARCore on Android with CameraX.

Using smartphone camera as a visual device to interact with the real world has becoming ever popular with utilization of deep learning and augmented reality as a tool. There is no doubt that Tensorflow Lite, the Tensorflow implementation for smartphone devices, including Android and iOS, is one of the most practical library to run inference in the devices. Running inference int the device, however, is not only thing you have to deal with for the device-side inference. You at least have to…

Shibui Yusuke

technical engineer of cloud, container, Kubernetes, ML, and AR.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store