The Apache Http Components client changed quite a bit from 3.0 to 4.0 beta. It wasn’t entirely clear to me from the documentation how to set request headers and cookie policies with the new client. After a lot of fishing around I was able to figure it out, here is the code for anyone trying to do the same thing.

Downloading a page with the 3.0 client:

HttpClient httpClient = new HttpClient();
 
GetMethod get = new GetMethod("http://www.twitter.com/teacurran");
 
// set a faux User-agent
get.setRequestHeader("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)");
 
// set the cookie policy
get.getParams().setCookiePolicy(CookiePolicy.RFC_2109);
 
int responseCode = httpClient.executeMethod(get);
System.out.println(get.getResponseBody());
 
get.releaseConnection();

here is the same thing with the 4.0 client:

DefaultHttpClient httpClient = new DefaultHttpClient();
 
HttpGet get = new HttpGet("http://www.twitter.com/teacurran");
 
// set a faux User-agent
get.addHeader("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)");
 
// set the cookie policy
get.getParams().setParameter(ClientPNames.COOKIE_POLICY, CookiePolicy.RFC_2965);
 
HttpResponse response = httpClient.execute(get);
 
int responseCode = int code = response.getStatusLine().getStatusCode();
HttpEntity entity = response.getEntity();
 
System.out.println(EntityUtils.toString(entity));