{"id":3031,"date":"2014-08-01T16:30:12","date_gmt":"2014-08-01T15:30:12","guid":{"rendered":"http:\/\/t-machine.org\/?p=3031"},"modified":"2014-08-01T16:30:12","modified_gmt":"2014-08-01T15:30:12","slug":"opengl-es-2-video-on-iphone-in-3d-and-texture-map-sources","status":"publish","type":"post","link":"http:\/\/new.t-machine.org\/index.php\/2014\/08\/01\/opengl-es-2-video-on-iphone-in-3d-and-texture-map-sources\/","title":{"rendered":"OpenGL ES 2 – Video on iPhone in 3D, and Texture-Map sources"},"content":{"rendered":"

This is a blog post I wrote 6 months ago, and my life got too busy to finish it off, finish testing the code, improving the text. So I’m publishing this now, warts and all. There are probably bugs. There are bits that I could explain better – but I don’t have time. Note that the code is ripped from apps where I ACTUALLY USED IT – so I know it does work, and works well!<\/span><\/p>\n

In brief:<\/p>\n

    \n
  1. Using different “sources” of textures in OpenGL, not just images!\n
  2. Using CALayer’s as a source — allows for video in 3D!\n
  3. Using “Video straight from the video decoding chip” as a source – faster video!\n<\/ol>\n

    <\/p>\n

    Procedural vs Mapped Textures<\/h2>\n

    Now you’ve made both Procedural Textures (Part 5) and TextureMapped Textures (Part 6), it’s time to see what they can each do.<\/p>\n

    The Real World versus The Digital World<\/h2>\n

    You can use a photo as a TextureMap, isn’t that great. Does it look like the thing you photo’d?<\/p>\n

    Not much. In reality, the object should have hilights on sharp edges, and have shadows in the cracks. It probably refracts light (anything even partially transparent – including human skin). Some parts of it might be fractal – e.g. the bumpiness of waves in the sea remains the same even as you zoom out. Some parts would be static – e.g. in a brick wall: the arrangement of mortar between bricks doesn’t reappear at smaller scale as you zoom in!<\/p>\n

    So … in an ideal world … we’d use Procedural textures for everything. We’d get realistic lighting, software control of every variable.<\/p>\n

    But the world is full of video and photos already – billions of them – and none of them.<\/p>\n

    So, to make something look realistic in 3D, you need to describe it completely in software.<\/p>\n

    Sources<\/h2>\n

    Every texture in GL has a source.<\/p>\n

    If it’s a texturemap, the simplest case: the “source” is a chunk of RAM on the GPU, labelled with an integer.<\/p>\n

    If it’s a procedural texture, the “source” is the GPU-program (Vertex\/Fragment Shader pair) that generates colours on the fly. Since the GPU program could reference anything as input data, it may in turn have sources that feed into it. A common setup is:<\/p>\n