acrane on "get_adjacent_post with multiple hierarchal categories"

I have a portfolio (CPT) that uses hierarchal taxonomies. Some projects are in more than one category. I want to be able to click to the next/previous project in a specific category (that i've passed in a url variable as the term ID).

get_adjacent_post() doesn't seem to work because it gets confused as some projects are in more than one category. But since I know the the term ID i want to use, how would I query the next/prev chronological post from the current viewed post?

I've even tried a custom query with the term id passed and next post id, which I thought should work, but it doesn't for some reason.

Here are my arguments:

$term_id = $_REQUEST['cat']; // from the url echoing a term id
$tax = 'portcats';
$termName = get_term( $term_id, $tax );

$args = array(
	'post_type' => array('portfolio'), //CPT
        'posts_per_page' => '1',
	'p' => $prev_post->ID,
	'tax_query' => array(
		array(
			'taxonomy' => 'portcats', // Tax
			'field' => 'slug',
			'terms' => $termName->slug, // slug
		)
	)
);