Gekkio's technical blog

Modern technologies with passion

Navigation Menu

Wandboard Quad

Posted on May 30, 2014 | 0 comments

I blogged last year about compiling Cyanogenmod for YW9300, but you might guess from the lack of updates that I never succeeded. The YW9300 had a lot of operating system problems, and I eventually lost interest in the device. The online shop w2comp.com even stopped selling the devices due to the problems, but it looks like they are back in stock now. The manufacturer might have fixed the problems, but I moved on to a different device for my personal Android hacking needs.

So, my current device is a quad-core Wandboard, which is a very nice little device with a Freescale iMX6 SoC. Freescale, Wandboard developers, and a company called Boundary Devices provide the source code of various different versions of iMX6 Android, which makes porting Cyanogenmod a doable task. Wandboard doesn’t have Android 4.4 yet, but I’m sure it will be released at some point! Overall I’m very happy with the device, and I find the overall software and hardware quality to be superior to most cheap Android devices in the similar price range.

One annoying thing about Wandboard is the lack of NAND memory, which means I have to boot using an SD card, which is obviously much slower. The quad model has a SATA port, but using a SATA SSD with Wandboard would require routing of power and a much bigger case than my current one. Using the SD card makes the operating system quite slow, so I might have to connect an SSD anyway at some point to be able to do anything meaningful with the device.

Read More

Android hacking with YW9300: Introduction

Posted on Jun 22, 2013 | 8 comments

I’ve spent a huge chunk of my free time this week on learning Android internals by tinkering with an Android device called YW9300. It’s a reasonably cheap device with a powerful SoC (system-on-a-chip) called Exynos 4412 (officially “Exynos 4 Quad”). This is the same chip that exists in every Galaxy S III phone, and is also used in some other Samsung and 3rd party products. My goal was to create a bootable custom ROM with Android 4.2 (“Jelly Bean”), because I was unsatisfied with the stock ROM supplied by the manufacturer (I’ll explain later why). I ended up failing my goal and bricking the device, but I thought I should share my experiences, so other people could use the knowledge I have gained, and avoid the mistakes I did. (Update: I thought I had already tried it, but I managed to recover using an original update.zip!)

What exactly is YW9300?

YW9300 is an Android TV box device from a Chinese manufacturer called YW Terminal. The device is marketed as a cheap multimedia device to be used with a TV, and it comes with XBMC and a media player preinstalled. The device includes a built-in 802.11n wireless adapter, but has no Bluetooth or Ethernet. At the moment of writing the price of the cheaper model with 1GB memory and 8GB internal storage was $120.

YW9300 front photo

YW9300 front

YW9300 back photo

YW9300 back

Exynos 4412 is a very powerful SoC at this price point, and the only real competitors are some Allwinner, Rockchip, and i.MX devices. There is a cheaper device (ODROID U2) with Exynos 4412, but I like the minimalistic approach of YW9300 and the fact that it already has a case. The only problem in my opinion is the software, which was a major disappointment.

Issues with the stock ROM

Android version

The manufacturer and sellers claim that it contains Android 4.2.1, but actually it is 4.0.x. This is easily verifiable by looking at the build.prop file that I will explain in a later post. Even the device itself says Android 4.2.1 in the About-section, but that is wrong and seems to be just the manufacturer trying to trick people. Claiming they have Android 4.2.1 and support for Flash should already ring the alarm bells, because Flash does not work on Android 4.2+.

USB camera support

The marketing material correctly mentions that there is no internal camera or microphone, but they still give the impression that external devices could be used for video chat (e.g. Skype). Based on my testing their kernel does NOT include the uvcvideo driver, which would be required for most USB webcams.

Hardware decoding

While Exynos 4412 supports hardware decoding of many formats, the MediaCodec API required for using such codecs in Android apps was added in API level 16 (= Android 4.1.x). The native hardware decoding libraries exist in the system, but as far as I know they are pretty much useless without the MediaCodec API. Claiming support for hardware decoding is most likely just a result of the Android version confusion mentioned above.

1080P freezing / HDMI problems

I haven’t seen these problems myself, but W2comp.com has stopped selling the boxes due to these issues. These might of course be issues with Samsung source code, which would then appear in custom ROMs too, but at least with a custom build there would be no need to wait for the manufacturer to fix the issues.

Cyanogenmod 10.1 as the custom ROM

The software wasn’t good enough for me, and this was a great opportunity to learn more about Android internals, so I decided to build my own custom ROM with Android 4.2.x, and all the needed features. I strongly recommend all serious Android developers to do some tinkering with custom ROMs, because it is an extremely useful learning experience.

At this point you might be wondering why I chose CM (Cyanogenmod):

  • It is the most popular custom ROM, and is actively developed and maintained
  • I’m already using it on my HTC One X, so I’m familiar with it
  • At the time of writing, CM 10.1 nightlies had experimental support for Exynos 4412 devices like Galaxy S III
  • The standard AOSP source code doesn’t support Exynos 4, and while the necessary source code is available from other sources, CM has already retrieved everything into their Github repositories

On the other hand, choosing CM did lead to some issues:

  • The Exynos 4412 code was still too much in flux, and mixed sometimes incompatible versions of third party libraries
  • The codebase is geared towards phones and tablets, while the YW9300 is a TV box. This made it a bit difficult to strip away all the phone-like stuff (radio, camera, gps, nfc, etc.)

The first issue was a major source of complications, because some libraries simply didn’t compile correctly due to too old dependencies. The second issue is getting better all the time, because CM 10.1 has introduced support for the ODROID-U2 development board. Too bad I initially missed this and used Galaxy S III code as the template :cry:

Still, even with these two issues I think CM was the right choice. Most other notable ROMs use CM as the base and just add some cool extra stuff, so I think it was simpler to just use the default CM code without any additions. In the next blog post I’ll take a closer look at the hardware so we’ll know what we are dealing with.

Read More

Increased compile-time safety with phantom types

Posted on Feb 7, 2013 | 0 comments

Introduction

Using phantom types is a very simple technique that can be used to increase the compile-time safety of code. There are a lot of potential use cases with different levels of complexity, but even a very lightweight usage of phantom types can significantly increase the compile-time safety.

A phantom type is simply a parameterized type with an unused type parameter. For example:

public class MyPhantomType<T> {
  public String sayHello() {
     return "hello";
  }
  // other methods/fields that never refer to T
}

This example class has a type parameter T, but it is never actually used in the code. At first glance this doesn’t seem to be very useful, but that’s not true! All object instances of phantom types carry the type information, so this technique can be used to “tag” values with some extra information that can be compile-time checked. We can of course escape the typing at any time by writing code without the generics, but that should be avoided at all costs. Some languages, such as Scala completely disallow dropping type parameters, so with Scala you would always have to keep the type information completely.

Example use case and implementation

One of the simplest and most useful use cases for phantom types is database ids. If we have a typical three-layer (data, service, web) Java web application, we can gain a lot of compile-time safety by replacing the use of raw ids with phantom types everywhere except at the “endpoints” of the architecture. So, the data layer will put raw ids to database queries, and the web layer might get raw ids from external sources (e.g. HTTP parameters), but otherwise we are always dealing with phantom types. In this example I assume that the database id type is always 64-bit long number. First we’ll need marker interface that will be implemented by all “entity classes”, which should be supported by the phantom type id mechanism:

public interface Entity {
  Long getId();
}

The only purpose of this marker interface is to restrict our phantom typed id to a certain set of tagged classes, and provide the getId method that will be used in the implementation. The actual phantom type is an immutable container for a single id value. The type parameter represents the “target type” of the id, which makes it possible to differentiate between id values of different entities in a compile-time safe way. I like to call this class Ref (shorthand for Reference), but this is just a personal choice.

@Value
@RequiredArgsConstructor(AccessLevel.PRIVATE)
public final class Ref<T extends Entity> implements Serializable {
  public final long id;  

  public static <T extends Entity> Ref<T> of(T value) {
    return new Ref<T>(value.getId());
  }
  public static <T extends Entity> Ref<T> of(long id, Class<T> clazz) {
    return new Ref<T>(id);
  }

  @Override
  public String toString() {
    return String.valueOf(id);
  }

}

This example class uses the @Value and @RequiredArgsConstructor annotations from Project Lombok. If you don’t use Lombok, add the constructor, getter, equals, and hashCode implementations manually (or look for the complete implementation below). Note how the type parameter T is never used anywhere. This also means that you cannot at runtime know the type of the Ref, but that is not usually necessary.

Using the example implementation

Now, we’ll replace the use of raw ids with Refs whenever possible. For example, we could have a service-level method that add a user to a group:

void addUserToGroup(long userId, long groupId);
// without parameter names
void addUserToGroup(long, long);

// VS

void addUserToGroup(Ref<User> userRef, Ref<Group> groupRef);
// without parameter names
void addUserToGroup(Ref<User>, Ref<Group>);

Now, when we want to call this method, we’ll always need Ref objects instead of raw long values. In this example there are two ways to get Ref values.

  1. If you have an instance of the actual object, call Ref.of(object). This is the most common method in layers other than web
  2. If you have a raw id, and you know the target type, call Ref.of(id, TargetType.class). This is usually required in the web layer if the raw id comes from an external source

In order to extract the raw id value from the Ref, you can read the field or use the getter. This is typically only needed right before database query construction.

Closing thoughts

In order to understand the benefits of Refs, try to think about the following cases:

  • What happens if you change the order of parameters in a method call which takes ids of different types? (for example our addUserToGroup)
  • What happens if you change the type of the database id (e.g. Integer -> Long, or Long -> UUID)?
  • How likely will you get runtime errors, if you often have method parameters of the same type as the id, but they are not ids? For example, if you have Integer ids and you mix ids and some sort of list indexes in the same method

In all of these cases, the use of Refs guarantees that you get a compile-time error in places where the code is not correct. In a typical codebase this is a huge win with very little effort. Compile-time safety decreases the cost and difficulty of refactoring, which makes maintaining the codebase much, much easier and safer.

Database ids are just a simple example of phantom types. Other typical use cases include some sort of state machines (e.g. Order<InProcess>, Order<Completed> vs just Order objects), and some kind of unit information for values (e.g. LongNumber<Weight>, LongNumber<Temperature> vs just longs).

Ref<T> implementation (without Lombok)

public final class Ref<T extends Entity> implements Serializable {
  public final long id;

  public static <T extends Entity> Ref<T> of(T value) {
    return new Ref<T>(value.getId());
  }
  public static <T extends Entity> Ref<T> of(long id, Class<T> clazz) {
    return new Ref<T>(id);
  }

  @Override
  public String toString() {
    return String.valueOf(id);
  }

  private Ref(long id) {
    this.id = id;
  }

  public long getId() {
    return this.id;
  }

  @Override
  public int hashCode() {
    return (int) (id ^ (id >>> 32));
  }

  @Override
  public boolean equals(Object o) {
    if (this == o)
      return true;
    if (o == null || o.getClass() != this.getClass())
      return false;
    Ref<?> other = (Ref<?>) o;
    return other.id == this.id;
  }
}

 

Read More

ZK Gritter: Growl-like notifications for ZK apps

Posted on Oct 22, 2012 | 4 comments

Introduction

Jawwa ZK Gritter is an open source library that can be used to add Growl-like notifications to ZK apps. The library provides a simple-to-use server-side Java API that can be used to control notifications in the application. Installation instructions are available in the reference manual.

A sample app that demonstrates all the customization options is available at Github:

https://github.com/Gekkio/blog/tree/master/2012/10/zk-gritter

Usage

The library provides a class called Gritter, which contains multiple static methods. Notifications are added using the builder pattern, so you must first obtain a notification builder, build the notification by calling the appropriate methods, and finally complete the builder and show the notification.

Gritter.notification().withTitle("ZK Gritter demo").withText(LOREM_IPSUM).
  show()

All notifications require both the title and text parameters, and you must remember to call show or the notification will not be actually shown to the user.

Customization

The library supports multiple parameters that can be used to customize the appearance and behaviour of the notifications. The API is fully documented with Javadoc.

Timeout and stickiness

By default all notifications fade out in 6 seconds, but this can be overridden by setting a custom timeout in milliseconds.

Gritter.notification().withTitle("ZK Gritter demo").withText(LOREM_IPSUM).
  withTime(500).show();

You can also add sticky notifications that don’t fade out at all unless the user closes them or they are removed programmatically.

Gritter.notification().withTitle("ZK Gritter demo").withText(LOREM_IPSUM).
  withSticky(true).show();

All visible notifications can be cleared manually. Calling removeAll removes both normal and sticky notifications.

Gritter.removeAll();

Custom CSS

The sclass parameter can be used to set a custom CSS class.

<style>
  .gritter-red .gritter-top {
    background-image: url(gritter-red.png);
  }
  .gritter-red .gritter-item {
    background-image: url(gritter-red.png);
  }
  .gritter-red .gritter-bottom {
    background-image: url(gritter-red.png);
  }
</style>
Gritter.notification().withTitle("ZK Gritter demo").withText(LOREM_IPSUM).
  withSclass("gritter-red").show();

Custom image

Notifications can also include an optional 48x48px image.

Java code:

Gritter.notification().withTitle("ZK Gritter demo").withText(LOREM_IPSUM).
  withImage("/warning.png").show();

Read More

Advanced ZK: Asynchronous UI updates and background processing – part 2

Posted on Oct 20, 2012 | 3 comments

Introduction

In part 1 I showed how server push and threads can be used to execute background tasks in a ZK application. However, the simple example had a major flaw that makes it a bad approach for real-world applications: it starts a new thread for each background task.

JDK5 introduced the ExecutorService class, which abstracts away the threading details, and gives us a nice interface which can be used to submit tasks for background processing.

In this blog post I will describe the most important parts of creating a ZK app, which contains a background task that takes a string, and returns it in uppercase. The complete sample project is available at Github:

https://github.com/Gekkio/blog/tree/master/2012/10/async-zk-part-2

1. Create an ExecutorService instance

First we need an ExecutorService that we can use in our ZK code. In most cases we want a shared singleton instance, which could be configured and managed by dependency injection (e.g. Spring). It is very important to make sure that the ExecutorService is created only once, and it’s shut down properly with the application.

In this sample project I will use a simple holder class, which manages the lifecycle of a single statically available ExecutorService instance. This holder must be configured as a listener in zk.xml.

package sample;

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

import org.zkoss.zk.ui.WebApp;
import org.zkoss.zk.ui.util.WebAppCleanup;
import org.zkoss.zk.ui.util.WebAppInit;

public class SampleExecutorHolder implements WebAppInit, WebAppCleanup {

    private static volatile ExecutorService executor;

    public static ExecutorService getExecutor() {
        return executor;
    }

    @Override
    public void cleanup(WebApp wapp) throws Exception {
        if (executor != null) {
            executor.shutdown();
            System.out.println("ExecutorService shut down");
        }
    }

    @Override
    public void init(WebApp wapp) throws Exception {
        executor = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
        System.out.println("Initialized an ExecutorService");
    }

}

Note that the thread pool is configured using a fixed size based on processors in the system. Proper thread pool sizing is very important, and depends on the type of tasks you intend to execute. The maximum number of threads is also the maximum amount of simultaneous concurrent tasks!

2. Write event classes that model the results of the background task

We’ll use ZK server push to communicate the task results back to the UI, so the results must be modeled as ZK events. It’s always a good idea to create custom subclasses of Event instead of adding the results in the data parameter, because a custom class is more typesafe and can supports multiple fields.

The first event class represents a status update that is sent while the task is still running. In this example it will contain the amount of characters in the input string.

package sample;

import org.zkoss.zk.ui.event.Event;

public class FirstStepEvent extends Event {

    public final int amountOfCharacters;

    public FirstStepEvent(int amountOfCharacters) {
        super("onFirstStepCompleted", null);
        this.amountOfCharacters = amountOfCharacters;
    }

}

The second event class represents the fully completed task. In this example it contains the input string in upper case.

package sample;

import org.zkoss.zk.ui.event.Event;

public class SecondStepEvent extends Event {

    public final String upperCaseResult;

    public SecondStepEvent(String upperCaseResult) {
        super("onSecondStepCompleted", null);
        this.upperCaseResult = upperCaseResult;
    }

}

3. Write the task class

The task class should have the following characteristics:

  • It implements Runnable
  • It takes all required input data as constructor arguments (the data should be immutable if possible!). This input data must be thread-safe, and generally should not include any ZK-related stuff (no components, sessions, etc.). For example, if you want to use a Textbox value as input, read the value in advance and don’t pass the Textbox itself as an argument.
  • It takes a Desktop, and at least one EventListener as constructor arguments. They are needed for sending the results back to the UI

In this example the only input data is a string that will be used to compute the task results.

package sample;

import java.util.Locale;

import org.zkoss.zk.ui.Desktop;
import org.zkoss.zk.ui.DesktopUnavailableException;
import org.zkoss.zk.ui.Executions;
import org.zkoss.zk.ui.event.Event;
import org.zkoss.zk.ui.event.EventListener;

public class SampleTask implements Runnable {

    private final String input;
    private final Desktop desktop;
    private final EventListener<Event> eventListener;

    @SuppressWarnings({ "rawtypes", "unchecked" })
    public SampleTask(String input, Desktop desktop, EventListener eventListener) {
        this.input = input;
        this.desktop = desktop;
        this.eventListener = eventListener;
    }

    @Override
    public void run() {
        try {
            // Step 1
            Thread.sleep(10000);
            Executions.schedule(desktop, eventListener, new FirstStepEvent(input.length()));

            // Step 2
            Thread.sleep(10000);
            Executions.schedule(desktop, eventListener, new SecondStepEvent(input.toUpperCase(Locale.ENGLISH)));
        } catch (DesktopUnavailableException e) {
            System.err.println("Desktop is no longer available: " + desktop);
        } catch (InterruptedException e) {
        }
    }

}

Note how all the constructor arguments are stored in private final fields, and how the input data is immutable (Strings are immutable in Java!). The task simulates long-running processing by using Thread.sleep, and submits a status event when the “processing” is half done.

4. Schedule tasks in ZK composers

Using the task in composers is very simple. You only need to enable server push, and submit a new task instance to the executor. This automatically starts the task once a free background thread is available.

desktop.enableServerPush(true);
// Get the executor from somewhere
executor = SampleExecutorHolder.getExecutor();
executor.execute(new SampleTask(input.getValue(), desktop, this));

In this sample the composer extends GenericForwardComposer, which implements EventListener, so it can itself handle the resulting task events. Both events are handled by methods that update the UI with status information.

public void onFirstStepCompleted(FirstStepEvent event) {
    status.setValue("Task running: " + event.amountOfCharacters + " characters in input");
}

public void onSecondStepCompleted(SecondStepEvent event) {
    status.setValue("Task finished: " + event.upperCaseResult);
}

Final words

It’s quite easy to add robust support for long-running tasks in a ZK application by using this technique. The resulting code in ZK composers is very simple, because the results are passed using the typical Event/EventListener paradigm that is very common within ZK apps.

The biggest dangers in this technique are thread-safety bugs, which can be very difficult to debug. It is absolutely crucial to fully understand the threads where every piece of code is executed, and ensure that all shared state is fully thread-safe. Using immutable input data, and immutable output events is usually enough to ensure safety as long as the background task itself doesn’t access other non-thread-safe resources. Some common mistakes are:

  • Invoking thread-local dependent library methods in the background task (e.g. any method that seems to magically get the “current” value of some type). The background threads will not automatically contain the same thread-local values as servlet threads, so by default all these kind of methods will fail. For example Sessions.getCurrent(), Executions.getCurrent() in ZK, many Spring Security static methods.
  • Passing non-thread-safe parameters to the background task. For example, passing a mutable List that might be modified by the composer while the task is running (always make copies of mutable collections!).
  • Passing non-thread-safe result data in events. For example, passing a List in a result event, while the List will be modified later on in the task (always make copies of mutable collections!).
  • Accessing non-thread-safe methods in the desktop. Even though you have access to the desktop in the background task, most desktop methods are not thread-safe. For example, calling desktop.isAlive() is not guaranteed to return the status correctly (at least in ZK 6.5 the method relies on non-volatile fields, so writes are not guaranteed to be visible in the background thread)
Read More

Java pitfalls: Field access in inner classes

Posted on Feb 28, 2012 | 1 comment

This is not a “pitfall” per se, but an implementation detail worth knowing. Let’s say I have a inner class with a field. Such a field is visible to the enclosing class, but which one of the following ways is the fastest way to access it? Note! I’m only looking here at the generated bytecode, and not considering any JIT optimizations, so this “performance analysis” is very naïve

public class Test {

  public void testAlternatives() {
    // Alternative 1
    System.out.println(new Inner().field);
    // Alternative 2
    System.out.println(new Inner().getField());
    // Alternative 3
    System.out.println(new Inner2().field);
    // Alternative 4
    System.out.println(new Inner2().getField());
  }

  class Inner {
    private int field;

    public int getField() {
      return field;
    }
  }

  class Inner2 {
    int field;

    public int getField() {
      return field;
    }
  }

}

An intuitive answer is that alternatives 1 and 3 are equally fast because the field is always visible to the enclosing class, and both use field access which is overall slightly faster than method access used in alternatives 2 and 4. However, there’s an implementation detail that causes this to be untrue. The JVM itself does not have a concept called “inner classes”. The whole concept is implemented by the Java compiler and in the bytecode level everything consists of normal classes.

The issue here is that if the inner class has a private field, and the compiler will eventually compile the inner class as a normal class. A private field in a normal class cannot be accessed by other classes, so the enclosing Test class cannot “see” the field without some tricks. Here’s the above code “desugared” to what the compiler actually compiles to bytecode:

public class Test {
  public void testAlternatives() {
    // Alternative 1
    System.out.println(Test$Inner.access$000(new Test$Inner(this)));
    // Alternative 2
    System.out.println(new Test$Inner(this).getField());
    // Alternative 3
    System.out.println(new Test$Inner2(this).field);
    // Alternative 4
    System.out.println(new Test$Inner2(this).getField());
  }
}

class Test$Inner {
  final Test this$0;

  private int field;

  Test$Inner(Test test) {
    this$0 = test;
  }

  public int getField() {
    return field;
  }

  static int access$000(Test$Inner inner) {
    return inner.field;
  }

}

class Test$Inner2 {
  final Test this$0;

  int field;

  Test$Inner2(Test test) {
    this$0 = test;
  }

  public int getField() {
    return field;
  }

}

As you can see, a package-level static accessor method called access$000 is generated in order to grant access to the private field. Now it’s easier to see that alternative 3 will most likely be the fastest one, because it is the only one that uses direct field access. Using package access in fields is a micro-optimization, but this whole thing is definitely a detail that should be known by Java developers. In performance-critical parts of code it might actually matter, and the Android performance guide actually mentions this implementation detail.

This implementation detail may also cause slight confusion when field access is attempted on a null reference of the inner class. Consider the following code:

public class NullTest {
  class Inner {
    private int field;
  }

  public void test() {
    Inner inner = null;
    System.out.println(inner.field);
  }

  public static void main(String[] args) {
    new NullTest().test();
  }
}

The variable “inner” is null, so a NullPointerException is obviously thrown. However, what is not apparent from the original code is that the exception is thrown inside the compiler-generated static accessor method!

$ java NullTest
Exception in thread "main" java.lang.NullPointerException
	at NullTest$Inner.access$000(NullTest.java:2)
	at NullTest.test(NullTest.java:8)
	at NullTest.main(NullTest.java:12)

The stack trace contains the intuitive exception source (line 8), but the real source will confuse developers who don’t know about compiler-generated accessor methods.

Read More