I believe if you actually want to browse the Active Directory, and not just read the authenticated user's profile, you need administrator consent for a web app. See http://msdn.microsoft.com/en-us/library/azure/b08d91fa-6a64-4deb-92f4-f5857add9ed8#BKMK_Graph
If you already knew that, then maybe it's a problem with how you've registered your app or the token itself. Make sure you've selected the appropriate permissions per that link in your app registration. If those look right, then you can check the token. There's a handy little token parser here: http://jwt.calebb.net/. Just paste in the value of your token and it will show you the decoded JSON. Look at the scope or scp parameters.
{
"typ": "JWT",
"alg": "RS256",
"x5t": "asdfsadfasdfsa"
}
{
"aud": "https://graph.windows.net/",
"iss": "https://sts.windows.net/<SOMEGUID>",
"iat": 1418158549,
"nbf": 1418158549,
"exp": 1418162449,
"ver": "1.0",
"tid": "<SOMEGUID>",
"amr": [
"pwd"
],
"oid": "<SOMEGUID>",
"upn": "admin@contoso.com",
"unique_name": "admin@contoso.com",
"sub": "askdljalsdfs",
"puid": "1003BFFD88937280",
"family_name": "Administrator",
"given_name": "MOD",
"appid": "<YOUR APP ID>",
"appidacr": "0",
"scp": "Directory.Read user_impersonation UserProfile.Read",
"acr": "1"
}