Views of UIImagePickerController
The standard image capture in API in the iPhone SDK is the UIImagePickerController. There is much discussion on the web about how this can be customized via subclassing, both from the viewpoint of technical feasibility, and from the viewpoint of being allowed onto the AppStore. It is generally accepted that going direct to private frameworks is unacceptable, even though this arguably can give a better user experience. Phanfare had their app pulled from their AppStore for using the PhotoLibrary private framework, and returned with a new version that instead customizes the UIImagePickerController experience.
I took an in-depth look at the view structure that the standard UIImagePickerController creates. The first is obtained by looking at self.view and is not particularly interesting. It consists of a UITransitionController and a UINavigationController and is clearly the meta-system for switching between the capture and preview screens.
The second view hierarchy is created by PLCameraController, and can be obtained by implementing willShowViewController in the UINavigationControllerDelegate. The first callback to this comes when UIImagePickerController wants to display a PLCameraController. The view hierarchy is much more interesting:
Several of these classes are undocumented, but that doesn’t prevent us from manipulating them with standard APIs. Experimentation showed that the UIImageView highlighted in red is the important one - it’s the camera preview pane. You can remove every other view, and just retain this one, and you will get a full-screen camera view. You can also move the TPPushButton (which represents the cancel button) out of the hierarchy and place it elsewhere on the screen. Doing the same with the TPCameraPushButton causes a crash when you activate the button, because the PLCameraController tries to modify the buttons, and fails because the view structure has changed.
You can also add further views (as Phanfare did) to overlay extra details. Clearly there is yet more scope for customizing other aspects of the hierarchy - this requires more experimentation, and we’ll report on this as we learn more.
November 11th, 2008 at 3:00 pm
Subclassing UIImagePickerController…
I’m trying to modify the UI that the user is presented with when they’re taking a picture. Specifically, I want to remove the “Take Picture” title bar and replace it with a custom graphic.
I can cr……
November 11th, 2008 at 3:16 pm
[...] The Air Source placed an observative post today on Views of UIImagePickerControllerHere’s a quick excerptThe standard image capture in API in the iPhone SDK is the UIImagePickerController…. [...]
November 11th, 2008 at 3:33 pm
[...] The standard image capture in API in the iPhone SDK is the UIImagePickerController. There is much discussion on the web about how this can be customized, both from the viewpoint of technical feasibility, and from the viewpoint of being .. More [...]
November 16th, 2008 at 9:49 am
Is the UIImageView really the cam preview? I onlly get the closed iris view when I grab its image. Or does one have to initialise the camera first?
November 18th, 2008 at 4:14 pm
I haven’t yet figured out how to grab the image, but it’s definitely the cam preview - if you remove just that view from the superview, then you get no camera preview at all.
November 19th, 2008 at 4:06 pm
Ben,
thanks for the info. I did my own “investigation” and came up with the same view hierarchy, although, I must say that the ImageView *is* the Iris. So far I have this: PLCameraView has four subviews:
0: ?;
1: is the UIImageView (iris);
2: ?;
3: all the extra stuff. So to get a full screen preview, with nothing else, you only have to remove that last view (the PLCropOverlay). I tried getting the content of view 0 and 2, but only got black screens so far.
November 19th, 2008 at 4:10 pm
sorry, view 0 (first subview) is the cameraPreview, but rendering its content leads me to only a black screen
November 19th, 2008 at 4:13 pm
I think we need to trade some code. I’ll get my code up later this week.
November 19th, 2008 at 5:59 pm
ok, I got it… I know why I’m only getting a black screen: it’s because renderInContext is limited to only render some parts, not all.
November 19th, 2008 at 7:48 pm
That’s where I got to - black screen with renderInContext, yet I could swear I was grabbing view 1, not view 0. I’ll have to check when I’m back on the mac.
November 20th, 2008 at 1:13 am
Rolando - you’re absolutely right. My apologies; I obviously got my ordering wrong when going from my code to a PNG graphic. I’ve corrected the blog post.
November 24th, 2008 at 10:36 pm
Ben, I finally was able to grab the screen
I’ll put my code online once I clean it up a bit.Btw, the iPhone camera sucks for scanning barcodes in real time 
November 24th, 2008 at 11:14 pm
Awesome. Look forward to seeing it! The iPhone camera sucks quite a lot, never mind just for barcodes!
November 27th, 2008 at 5:04 pm
Ben, here’s a proof of concept: http://www.youtube.com/watch?v=uc-p221SQ0g, code to follow soon…
December 8th, 2008 at 6:18 pm
Rolando, any code to share yet??
December 10th, 2008 at 12:36 am
Amazing, thank you so much.
December 12th, 2008 at 12:06 am
I’m on the same track (just finished playing with the various camera picker subviews and layers before I found this page), if I get any code to work soon that manages to read the preview image I’ll post as well.
It would be a very nice thing to take to an iPhone Dev Camp I’m attending this weekend…
December 18th, 2008 at 3:02 pm
Any thoughts on how to use this to capture the event generated when the shutter button is pressed? I’d like to be able to trigger a function when the view changes to the preview view…
December 18th, 2008 at 3:07 pm
Tom - Sure. Add an action to the TPCameraPushButton.
December 18th, 2008 at 3:45 pm
It would be great to see some demo code…
December 18th, 2008 at 9:59 pm
Sweet, that works nicely. Thanks…
December 18th, 2008 at 10:01 pm
Tom - great
December 22nd, 2008 at 12:02 pm
Hi.
is it possible to grab a frame from incoming video and access the raw RGB data of the frame with this approach ?
December 22nd, 2008 at 6:40 pm
It’s certainly possible, the questionis whether you can do it with a legal API or not. Rolando says it’s possible, and we’re hoping to see some example code from him at some point.
December 23rd, 2008 at 2:47 am
Yes, it is possible with hacked toolchain, but it’s not desirable.
If we can get the raw image data through the official SDK, it will be great.
I am really looking forward Rolando’s sample code.
January 3rd, 2009 at 10:37 pm
I’ve been trying to get the pictures from the preview with certain success. Now, in the last 5 days, I’m stuck with the black pictures problem. Rolando says its possible but didn’t provide any more details. Could anyone tell me how to do it? Or can anyone tell me the email of Rolando? Email me at alexmipego @ gmail.com if you have any information please.
January 7th, 2009 at 10:20 pm
I am also getting black screens using renderInContext.. any help would be great rolando? For anyone that’s interested, you can use [UIImage imageWithCGImage:UIGetScreenImage()] but that is not an official SDK API call so it probably won’t fly in the app store (who knows)… I’m still looking for a way to do the screen capture with all official APIs.
January 8th, 2009 at 2:01 am
Hi.
Now i am writing the program for getting the image data from camera in real time, but have not a good way get it through the official SDK. Rolando, may i see your sample code?
Please send my mailbox. Thanks.
January 9th, 2009 at 3:45 pm
Hi.
I have the same problem to get a screen image from UIImagePickerController.
Obviously only Rolando knows how to get the images. Has anyone a response
from him? It would be great to know what the problem is. Thanks
January 12th, 2009 at 6:48 am
I have a small xcode project and some write up to demonstrate some tweaks to the camera interface here: custom UIImagePickerController camera view.
It looks like there might be a mistake on your diagram. While the camera view is the first subview of PLCameraView, it is a UIView, not a UIImageView.
January 14th, 2009 at 2:25 am
Hi, Iajos.
How can I get a screen image from PLCameraView?
Thanks.
January 15th, 2009 at 10:28 pm
@KUOLEI - search for the renderincontext method
@ALL - Anyone had an anwser from Rolando?
January 16th, 2009 at 3:27 pm
@kuolei & Alexandre
I have tried the renderInContext method with the PLCameraView but the
result is a black screen image. Does anyone knows the trick to get the right
image?
January 16th, 2009 at 3:50 pm
I was periodically in contact with Rolando, but I’ve had nothing but radio silence recently (or, indeed, not so recently given this thread has been running for over two months).
January 16th, 2009 at 11:10 pm
Is it possible that this has been blocked since v2.2? Have we seen any confirmation that anyone ( Rolando? ) has had it working since 2.2 came out?
January 17th, 2009 at 8:27 pm
The black screen image is the reason most people are here
Only rolando says he has been able to work around it. Although I suspect he used the UIGetScreenImage method which isn’t approved by Apple.
January 17th, 2009 at 9:17 pm
I asked Rolando by email if he used that method, and he indicated clearly that he had not. UIGetScreenImage is a bit slow in any case.
January 19th, 2009 at 7:48 pm
Ben, could u give me Rolando’s contact? Or asking him to contact me?
January 19th, 2009 at 7:51 pm
@Ben
The reason I ask is because I’ve something very cool ready except for the black screen image. I’m willing to share code or even release it opensource if I could get it working.
January 19th, 2009 at 7:57 pm
@Alexandre I’ve emailed Rolando directly with your email address and asking for an update. Rolando didn’t choose to make his email address public, so I can’t divulge it. Sorry.
January 20th, 2009 at 12:27 am
I understand. Im usually not this anoying on the internet, but Ive spent 2 weeks trying to figure out how to solve the problem. In fact, I know what Rolando means in his last post about the renderincontext not catching everything… I just can’t find a way around it. Lets hope he chooses to help me
Thanks.
January 20th, 2009 at 11:01 am
@Alexandre
I also stuck on this problem about the renderincontext since few weeks.
If you heard something from Rolando maybe you can share this feature
with us.
Thanks.
January 20th, 2009 at 6:30 pm
[...] The Air Source » Blog Archive » Views of UIImagePickerController The standard image capture in API in the iPhone SDK is the UIImagePickerController. There is much discussion on the web about how this can be customized via subclassing, both from the viewpoint of technical feasibility, and from the viewpoint of being allowed onto the AppStore. (tags: design workaround iphone UIImagePickerController) [...]
January 20th, 2009 at 7:37 pm
Well, I’ven’t heard from Rolando. I do however think that sharing what I think the problem might produce some ideas and maybe someone here can solve the problem.
So, the problem as I guess it is, is that the renderincontext message doesn’t work in all cases. This is poorly documented by Apple, they only say this:
«The Mac OS X v10.5 implementation of this method does not support the entire Core Animation composition model. QCCompositionLayer, CAOpenGLLayer, and QTMovieLayer layers are not rendered. Additionally, layers that use 3D transforms are not rendered, nor are layers that specify backgroundFilters, filters, compositingFilter, or a mask values. Future versions of Mac OS X may add support for rendering these layers and properties.»
Now… I’ve tried to figure out if any of those things were beeing used. The only way I know to do this kind of research is to use the debugger and inspect the properties (which, iirc doesn’t provide a 1 on 1 relation with the things specified by Apple). Since I am new to Cocoa/Objective C maybe there’s a better way to inspect the “tree” and properties?
Also, it occured to me that *maybe* one of those things (ex: composingFilter) could exist not on the preview UIView but on one of the parents. My guess on this is that since I didn’t found one on the UIView and the image is shown resized… then it could be the whole reason for our problems.
I think thats what I’ve to share. Please comment and provide details if any of you find a solution
January 21st, 2009 at 3:49 pm
Just to let everyone know: I have figured it out, but unfortunately, I can’t release any source code until the app for which the library (realtime barcode scanning) gets released, this is due to a nasty NDA with the client for which the app was made. FYI, I did not use UIGetScreenImage neither renderInContext, it was a pretty simple trick after all. Alexandre: you can inspect things using objective-c introspection (I also sent you an email)
January 22nd, 2009 at 1:58 am
Hi, Rolando.
I get the image data in realtime without renderInContext. But this image data is not a general image, the description of class is CAImageQueue in debugger. How can i get the pixel data for this “image”.My email is kuolei@gmail.com.
please give me an suggestion.
Thank you.
January 29th, 2009 at 4:43 pm
Hi kuolei. You’re right that what you get in the contents is a CAImageQueue. I’m not sure what Rolando has done.
However the only way I’ve found so far to get image data out of the CAImageQueue, which is not a public API, is to dig down into its internals and find a CoreSurface buffer attached to the camera device. Unfortunately, to use this you then need to call CoreSurfaceBufferLock() and friends, which requires linking the CoreSurface framework. It’s not enough to just do the equivalent of CoreSurfaceBufferGetBaseAddress() and copy (neglecting potential tearing), since the buffer is actually owned by the camera device and is not available to you without an IOKit call to map it into your process (which is normally done by CoreSurfaceBufferLock()).
For this reason I suspect Rolando will have followed a different path.
February 2nd, 2009 at 10:40 am
I just saw a post over at StackOverflow
http://stackoverflow.com/questions/332120/iphone-video-buffer - where a guy says he’s successfully used
image = [window _createCGImageRefRepresentationInFrame:rectToCapture];
Obviously a private API, but we’re trying it out now for the purposes of research and will report back.
February 3rd, 2009 at 11:56 am
_createCGImageRefRepresentationInFrame does appear to work, but it appears to be no faster and thus no better for us than UIGetScreenImage().
February 4th, 2009 at 9:26 am
Odd, I posted here yesterday but my browser ate it.
So far not useful but very interesting (and fun) the UIView -> CALayer -> contents (where contents’ class is a CAImageQueue), so CAImageQueue is a CGImage. This allows you do do fun things like create multiple active live video previews:
UIImage *image = [UIImage imageWithCGImage:(CGImageRef)view.layer.contents];
pipImageView.image = image;
Unfortunately, I cannot extract the bitmap from the CGImageRef (probably because as mentioned it is direct CoreSurface writes).
Core Animation also takes a snapshot of the video feed when doing something like a UIViewAnimationTransitionFlipFromRight. I have not been able to figure out a way to take advantage of that with a custom animation though.
Finally, I just found a great write up here:
http://translate.google.com/translate?prev=_t&hl=en&ie=UTF-8&u=http%3A%2F%2Fcntt.tv%2Fnodes%2Fshow%2F492&sl=vi&tl=en&history_state0=
February 12th, 2009 at 4:58 pm
Is there any way to take a closer look at any piece of code that was published on youtube by Rolando? I’m new in real-time application development but I hold a great interest in mobile augmented reality and any ways how to overlay full screen camera view with any kind of objects as labels/textviews or so.
For me also it was a great challenge to see something like this http://www.techcrunch50.com/2008/conference/presenter.php?presenter=71#video
I dont really know if the guys from tonchidot are using any private APIs or undocumented call or whatever. It will be really interested to discover at least some simple code examples and the way how to do it.
So if somebody can help me out on my investigations, it would be really appreciated.
February 13th, 2009 at 12:58 pm
Does anyone know what subview the activity indicator is when the user takes a photo? I would like that to be hidden as well
February 21st, 2009 at 5:48 am
Has anyone else figured out Rolando’s solution for? Could we have another hint?
It seems like getting the CGImage from preview.layer.contents is a step very near to a solution. I’d like to throw my email in the ring if anyone has a suggestion they’d like to pass along: socrates1024@gmail.com
February 23rd, 2009 at 1:07 am
Has anyone had any luck trying to mimick the likes of ‘QuadCam’ or ‘25 shot!’ ?
I’ve seen a few of these apps and they all appear to make the “shutter noise” for every frame which is captured, which is what lead me to believe they were synthesizing touch events, and the OS was triggering the sound.
Anyone have any ideas?
March 2nd, 2009 at 7:05 pm
Codz, QuadCam uses UIGetScreenImage() - I’ve been in contact with the developer.
Has anyone found an alternative to UIGetScreenImage / renderInContext ?
March 4th, 2009 at 12:02 am
Great thanks Ian, I thought as much… I’ve tried for ages trying to find an alternative.
March 10th, 2009 at 12:49 pm
Has there been any word from Roland? I’m really interested in any suggestions on how to get the preview image as well.
cmgreening at gmail.com
March 10th, 2009 at 1:38 pm
Has anybody had any success getting an app that tweaked PLCameraView approved by Apple? I’ve seen apps that clearly do, but ours got rejected…
March 11th, 2009 at 2:20 am
I can get pixelbytes from each images of preview as below code.
http://github.com/norio-nomura/iphonetest/blob/9713242dda6c6bc897da4bd639a1fdadc29b6fd7/CameraTest/Classes/CameraTestAppDelegate.m#L35
It uses undocumented methods, so Apple will not approve, but work well.
March 21st, 2009 at 7:57 pm
@Tom:
You tweaked the view structure and your app got rejected? Can you share the details? What did you do, and what did Apple say?
March 26th, 2009 at 10:14 am
Any word from Rolando?
I’m starting to see a ton of proof of concept augmented reality apps - so people are definitely managing to access the video stream.
I really want to add this feature to my Sudoku Grab application (http://www.youtube.com/watch?v=oImMJ6p6mKE). I can definitely make my puzzle detection and solver code work in real time so I could superimpose the solved puzzle on top of the image.
At the moment there’s no point in doing the optimization as it works fast enough (in the latest version <1second) for non real time.
If anyone can help them drop me an email chris at cmgresearch.com
There’s a promo code for my app in it (worth a huge $1!)
March 27th, 2009 at 6:18 pm
@Tom, ours did also.
Does anyone have any experience getting an app accepted that hides the “Take a Picture” overlay or changes its text?
The feedback from Apple seems to tell us that touching the PLCameraView object at all is against the rules.
March 30th, 2009 at 8:17 am
Yes mine just got rejected too since I called the “undocumented API:s in UIImagePickerController” (I am just hiding the ugly labels).
There are indications in the 3.0 release notes that the camera view is being changed, so maybe they are getting tougher on this issue.
Could we hope for a better API in later versions of 3.0? (but that is probably overoptimistic).
At least everyone show add a request for a better API in the bugreporter.
March 30th, 2009 at 8:46 am
Clearly some apps have been rejected - but there are plenty of apps out there that mess with the views, QuadCamera in particular - and also Microsoft Tag Reader. QuadCamera doesn’t seem to have any issues getting accepted, they are up to something like version 1.8 now, suggesting that they have been through the testing process 9 times.
March 30th, 2009 at 7:29 pm
Hi rolando,
I just want to confirm are you able to grab the screen after displaying the camera preview on it?
In my application I want video stream from camera before displaying it on screen. I will process that stream and then display on screen. Is this possible?
You can mail me at vishalnandanwar@gmail.com
Vishal N
May 23rd, 2009 at 8:57 am
Has anyone made any progress in obtaining raw image data from the camera? I know Rolando has successfully achieved this, but we’ve yet to hear from him.
@Rolando - Are you able to share your methods now? Also, did your app get rejected from the App Store? Could you share you experience with us?
Also, for those of you who have published apps (or attempted to) that have manipulated the PLCameraView, could you please share your experiences with Apple? Has anyone figured out a way to do this without getting rejected by Apple?
As Ben has mentioned, there are obviously people out there getting away with this. I would hate to see all my hard work go to waste b/c of a moody app reviewer. Somehow I feel like the only way to slip these apps through the reviewers is to make friends with one of the evangelists over in Cupertino.
Has anyone made contact with these guys? Furthermore, has anyone gotten specific feedback regarding policy on manipulating PLCameraView?
My implementation provides a subclass for UIImagePickerController and does not use any “undocumented” API’s so to speak. It seems ridiculous to me that these apps are getting rejected for hiding UI elements using calls like:
[[[view subviews] objectAtIndex:0] setHidden:YES]
Thoughts anyone?
June 10th, 2009 at 2:33 am
[...] did not discover everything on my own. I am using some information from lajos, The Air Source, and Erica [...]
June 18th, 2009 at 7:43 pm
[...] as it was very foreseeable indeed that mucking around with the internals of UIImagePickerController as described here would be likely to break with a system revision. [...]
June 20th, 2009 at 4:53 pm
Hi all
We have been browsing around for some time looking for info about the problems your discussing.
Today we came across a newly updated app: RedLaser, a barcode reader.
In their newest version on AppStore they have realtime scanning of the image from the camera, with custom overlays. Has anyone found new possibilities in latest SDK 3.0? Looks like they found a great approach in RedLaser, which passed the appstore review.
June 24th, 2009 at 5:19 am
Hey is this hierarchy is same for all the iphone os version!!! I had iphone 3.0 os my hierarchy is different from yours!!! what can i do with iphone 3.0??
June 24th, 2009 at 1:13 pm
No. The hierarchy has changed in iPhone OS 3.0. And remember that the behaviour is different on the 3GS, which has auto-focus, than on older devices.
July 7th, 2009 at 2:29 pm
I was playing around with PLCameraController. After updates to 3.0 same source code doesnot work. Not yet figure out the issue. Any ideas?
September 10th, 2009 at 7:37 pm
Well the new 3.1 OS certainly makes it a lot nicer to change the UI for the UIImagePickerController
OverlayViewController *oController = [[OverlayViewController alloc] initWithNibName:@”OverlayView” bundle:nil];
oController.tabController = self;
self.overlayController = oController;
[oController release];
UIImagePickerController *uip = [[[UIImagePickerController alloc] init] autorelease];
uip.delegate = self;
uip.sourceType = UIImagePickerControllerSourceTypeCamera;
uip.showsCameraControls = NO;
uip.cameraOverlayView = overlayController.view;
Currently trying to use the takePicture method in a timer to feed a barcode reader. takePicture seems to be taking a picture though it’s a 2MP (3G iPhone for ref), I resize that to 320×480 and feed it to the barcode reader but then it won’t recognize it. For some reason UIGetScreenImage() works really well though. I would like to stay away from using that, so any suggestions?
November 6th, 2009 at 2:28 pm
Has anyone figured out a solution to this trouble? Rolando, if you have a legal way of doing this or anyone else that is reasonably fast I would be willing to pay for the solution. The new 3.1 routines are too slow. UIGetScreenImage() works fast enough for our usage and uses little memory but it is part of the private api. email: delay at pobox dot com if anyone has any code they want to give away or sell that solves this trouble.