Returning IQueryable<T> rather than List<T>
Usually when I write repositories, I end up with code like this:
IList<Order> orders = repos.FindOrdersByStatus(OrderStatus.Packed);
One problem with this is that orders might contain 10,000 records, and I don't really want to pull all that back from the database in one go. What I think I want is this:
IList<Orders> orders = repos.FindOrdersByStatus(OrderStatus.Packed)
.Skip10)
.Take(10)
.ToList();
FindOrderByStatus is returning an IQueryable rather than IList. The beauty is that every query can be paginated for "free" - no need to modify repository method signatures with pagination fluff. And, pagination is executed in the database yet controlled by any client using the repository class.
Is anyone doing this?
Comments
-
I’ve tried something like this… the only catch I can think of is that if repo.FindOrdersByStatus() returns an IQueryable<>, then it is possible to forget you’re dealing with a db attached to that IQueryable and do something on it that IQueryable<> supports and yet your db provider does not.
It’s a minor detail and one that can be mitigated. I like your approach overall.
-
Thanks @Sean. Yeah, I was thinking about this. I’m not even sure I like the non visible boundry between LINQ-in-memory and LINQ-to-[insert provider here].
-
Tobin,
I’ve also been blogging about my experimentation with using Linq in my repositories and I’ve generally found that exposing IQueryable from your repositories adds some complexity issues in managing the differing limitations of the underying provider.
Specifically i’ve been using L2S and the Entity Framework, and their support of IQueryable varies somewhat. This really seems to be a symptom of IQueryable being a bloated interface, which I noticed you’ve blogged about recently :)
For me, the purpose of the repository is to obtain a reference to the root of an aggregate. A repository will therefore behave like a collection of Entities. Contrary to this, repositories that return IQueryable are returning a query object. I think this probably goes against the original purpose of a repository. What do you think?
This topic is summed up quite well by Fredrik Normén in his blog here: http://weblogs.asp.net/fredriknormen/archive/2008/04/24/what-purpose-does-the-repository-pattern-have.aspx
Anyway, I just wanted to say i’ve really enjoyed reading your blog, so keep the good posts coming! :)
-
My main approach has always been to ensure that the SQL execution never happens outside of the repository boundary. So returning an IList is probably the safest.
An alternative which I have used previously is to return IQueryable but ensure that it has been executed – e.g return (original db IQueryable).ToList().AsQueryable(). Bit of a performance hit but allows linq based post processing on the repository result.
You might also like to check out – http://code.google.com/p/linq-specifications/ where I touch some of these concepts (my current pet project – FTW!)
-
@Craig
Thanks for the comment. That’s an interesting link too with good discussion.
You’re right, returning a query object is significantly different to returning a list of entities.
I don’t like issues introduced by IQueryable as I mentioned in my ramblings about
IAlmostQueryable!? It’s a subset of IQueryable that supports only paging. It’s still deferred execution though, and still returning a Query Object. And still far from perfect :)I’m now having a re-think…
-
@Steve
Can see your point about deferred execution. It’s open for potential abuse. But, I do wonder if the benefits outweigh the drawbacks.
Specifications seem like they might be the answer. But, where do the specifications live? A Repository is nice because it’s a well known location for queries against a given aggregate root.
-
@Tobin
I agree with Steve that SQL execution should never happen outside of the repository boundary, but would suggest that the benefits of IQueryable maybe better off being encapsulated completely within Repository.
Currently, I favour a repository interface that looks like this:
T Get(ISpecification<t> criteria); bool TryGet(ISpecification<t> criteria, out T entity); IList<t> Find(ISpecification<t> criteria); IList<t> Find(ISpecification<t> criteria, int page, int itemsPerPage); long Count(ISpecification<t> criteria); bool Exists(ISpecification<t> criteria);
In regards to the location of specifications, I generally believe them to be a domain concept, containing things like GoldCustomerSpecification and the like. I’m also currently considering whether paging maybe wrapped up in its own PagedItemSpecification…
It’d be great to get your thoughts on this!
-
@Craig
I like your solution.
Do you find you can now have a generic repository because you don’t need finder methods on your repositories? By that, I mean you don’t ICustomerRepository, IOrderRepository etc. You just have IRepository<customer> and IRepository<order>?
So you put them in the domain? Do you just plonk them alongside other domain artifacts (entities etc), or stick em in a “Specifications” namespace?
I was thinking about specifications this morning. The problems remaining are that
a) It feels quite heavy having to create one class for every query. But this is just a “feeling”, I haven’t tried it.
b) If I don’t do that, then I’ll end up mixing finder methods with specifications. I strive for one-size-fits-all solutions, so I’d rather not do this :)
c) The client code might not be so clear to newcomers to the code base.
var customers = repos.FindGoldCustomers();vs
var customers = repos.Find( new GoldCustomerSpecification() );Actually, that’s not too bad. I wonder how it would look with a more complex query, with pagination…
var customers = repos.Find( new GoldCustomerSpecification{Skip=10, Start=10, OnlyIncludeRecentBuyers=true});Hmmm, is this the kind of thing you have?
-
Exactly right, since I no longer need specific finder methods to retrieve the entities I need, I use generics in my repositories, specifically:
IRepository<taggregateroot> where TAggregateRoot : IAggregateRoot
In general I put most of my specifications alongside other domain artifacts, but only where the specification represents a domain concept. In the case of the GoldCustomerSpecification, it belongs in the domain since there is a “gold customer” in our ubiquitous language. This may be “all customers who’ve spend over £250 in the last month are considered to be gold customers” or something to that effect. I see these kinds specifications as value objects, since they seem to fit Evans description of a value object:
“An object that represents a descriptive aspect of the domain with no conceptual identity is called a VALUE OBJECT. VALUE OBJECTS are instantiated to represent elements of the design that we care about only for what they are, not who or which they are.”
I do also have some specifications that don’t exist in my domain, and these generally exist in my application layer. For example, to represent a more comprehensive set of (optional) search options for finding a customer, by name, number of orders, or whatever i’d have:
CustomerSearchCriteria : Specification<customer>, ISearchCriteria<customer>
These specifications may internally be formed of a combination of domain specifications (like GoldCustomerSpec) with other adhoc criteria (like name=”bob”).
In this example since ISearchCriteria<tentity> is a specialisation of ISpecification<tentity>, my repository can be simply be called from my application service like this:
repository.Find(customerSearchCriteria, pageIndex, itemsPerPage);
Does that make sense?
-
+1 to generic repository implementations. And +1 to this dicussion!
Personally, I find that the specifications (when properly named) adequately signify the intent of the operation so multiple repositories are not necessary.
You can always go down the extension method route if you want to tidy up the api. For example:
public static IQueryable<customer> FindGoldCustomers(this IRepository<customer> repository);
Also, If you are worried about creating a specification for every single query – I agree with you. Anything remotely complex should be wrapped up but I feel the maintenance overhead makes it pointless for the most basic of queries. Check out my solution at – http://www.mostlyclean.com/post/2008/08/Linq-Expressions-The-Specification-Pattern-and-Repositories-Part-3.aspx.
-
@Craig
Thanks for elaborating on that! All makes sense.
- Generic repositories
- Specification value objects in domain for meaningful domain concepts
- More specification objects in the app layer for flexible customizable searches
- Specifications can be composed of other specifications
All sounds good, and covers all the bases I think. I’m guessing your specifications are all using Linq (given EF and L2S). I’m interested to see how doable this with HQL or ICriteria without exposing NHibernate outside of the data access layer.
I love the idea of a composite specification (not sure if it’s a true composite), where you can reuse specifications.
Thanks again for the comments.
-
@Steve
Interesting series of blog posts you have there! I get what your saying about some specifications being too trivial to justify. Ad-hoc specifications is a nice way around (and a nice phrase).
I see you’re using Linq for NHibernate. I’ve hit issues with getting some queries to work, have you had any troubles?
I usually write queries in HQL, and then come back and attempt to refactor them into Linq (I usually have about a 60% success rate!). Being able to mix and match query language with NHIbernate is awesome.
I’m interested in having a specification object that doesn’t expose Linq, where the client doesn’t know what query language is going on behind the scenes. This would let me use SQL, ICriteria, HQL, or Linq without clients knowing. I guess this is gonna be more tricky…
-
Not a fan of generic repositories (DAO’s), they seem cool at first but I don’t think they’re a good investment. Your repositories are supposed to help encapsulate all details of persistence not just querying. Without them how do you say that aggregate X is never deleted, or that its not deleted its archived, or that aggregate Y is built up from multiple data sources, or that query Z is very expensive and needs to be done using SQL. Those are all things I’ve had to be involved in when working with a moderately complex domain model and repositories helped encapsulate those details.
Maybe for simple green-fields domains I’d use a generic repository, but even then I find having real repositories far more useful.
Specifications themselves are definitely useful though, and Linq can make that easier, but the whole Adhoc approach is not one I like. We tried that quite a while ago when 2.0 came out but those qd hoc specs just speak to the language of the implementation not the language of the domain and so (in my view) are best used outside the domain.
@tobin You can definitely have non-Linq specs, but then you need to convert them into SQL/ICriteria or whatever which has to happen outside the specification itself. Doable but not a lot of fun.
-
Thanks for sharing your view, Colin.
That’s one downside to generic repositories I guess, they close the door to flexible decisions for each query.
Obviously you can use IoC and decorators for adding additional responsibilities, but not on a per query basis.
Another option is to mix both generic repositories with non-generic ones, but I’m looking for consistency, so mix n’ match isn’t favourable.
The thought of translating specifications into HQL etc is scary :) I’ve read the advice in DDD a few times, which says you have specifications that express constraints in terms of objects (i.e. Linq) and have them express the constraint as an SQL query or fragment. Still feels like too much work to do for all specifications.
Still looking for that simple solution :)
-
Another option is to mix both generic repositories with non-generic ones, but I’m looking for consistency, so mix n’ match isn’t favourable.
Agreed, which leaves me using custom repositories. I think thats fine though, they’re easy to implement and add to the expressiveness of the domain so I like them a lot.
The thought of translating specifications into HQL etc is scary :) I’ve read the advice in DDD a few times, which says you have specifications that express constraints in terms of objects (i.e. Linq) and have them express the constraint as an SQL query or fragment. Still feels like too much work to do for all specifications.
Definitely. Pre-linq I’d say 75% of the specs I wrote didn’t hit the DB, they were just for in memory and so easy to implement (IsSatisfiedBy(entity)). Any custom queries were implemented in the repositories which could do whatever they wanted (ICriteria/HQL depending on query complexity, SQL if required).
So that left a very few repository queries that took the specifications that were converted to ICriteria themselves, mainly for custom filtering and so on. It wasn’t nice converting from the spec to ICriteria and it wasn’t generic so you had to do it for each spec but it wasn’t a massive amount of work. Now though, especially when Linq to NHib is done, I’m thinking that I’d handle those cases by passing in custom Linq based SearchCriteria objects (which are kinda specifications).
Thats my take on the current situation anyway.
-
Cheers Colin, interesting indeed. How do you do pagination, btw? Do you have each finder method on the repository take a PageNo and A PageSize?
-
@Tobin
All my specifications are currently made up of Linq expressions and these are implemented in a similar manner to those in Steve’s blog. I use expressions since they can be used to express my domain logic soley in terms of my domain model, and can be applied to data irrespective of whether the data is an in memory collection, or from a Linq Provider.
As Colin points out, you can definately have non-Linq specs (assuming your repository implementation only makes use of an interface similar to specification.IsSatisfiedBy(entity)), however you’ll then need to convert them into SQL/ICriteria somewhere outside of the specification. Yves Goeleven has a good example of how this may be achieved, using a DataMapper, which can be found here:
http://www.goeleven.com/blog/entryDetail.aspx?entry=84
(Yves also has an sample implementation and provides both an AdoRepository and a NHibernateRepository)
In my repository implementations (such as a Linq to Sql implementation) I can add custom persitence related logic, such as “aggregate X is never deleted”, or “that its not deleted its archived” etc, and this persistance logic solely exists in the implementaion of the repository. I’ve also created a mechanism to implement fetching strategies, so specific entities can be pre-fetched where required. This allows me to optimise my data retrieval for specific use-cases.
All that said, it may make sense to favour using specific repositories that inherit from a generic repository as a “starting point” for repository implementations. That way you can implement optimisations as and when you feel they are necessary. For example:
CustomerRepository : DefaultRepository<customer>, ICustomerRepository
All this does is cut down on the (already pretty simple) task of implementing custom repositories.
I definately agree with Colin that Adhoc specifications are best left outside of your domain. As I mentioned before, I tend to use this kind of technique solely in my application layer, for application specific use cases.
@Colin
I currently using custom Linq based, application layer SearchCriteria objects (like CustomerSearchCriteria) that are in fact specifications in the form ISpecification<customer>. These SearchCriteria objects define a Linq Expression that represents the search. I’ve found them very effecive in communicating application specific filters in a flexible way. I’m currenly working on an example of this to post on my blog – it’d be great if I could get your feedback on it when its complete :)
-
@Craig Yup sounds good, will definitely give it a look over.
Sorry for not reading closely enough on Ad hoc specs, you did indeed say they were outside the domain! :)
@Tobin From what I remember pagination became just another criteria that you could add to the search criteria before sending it off to the repository. Client wise it worked a treat, as I say though in pre-Linq days the code to convert to ICriteria was not nice.
-
Although I added my initial thoughts to this post, I had a few additional thoughts on my journey home from work, and I’ve had a bit of a re-think on some of my views, so i’ve created a summary post on my blog. It’d be great to get some feedback on this – the post can be found here:
http://craigcav.wordpress.com/2008/10/22/what-purpose-does-the-repository-pattern-have/
-
@Craig
Thanks for the links re non-linq specs. I’m going to check those out…
Generic + Specific sounds like best of both worlds – and I’ve seen this done in some projects on CodePlex/Google Code.
Gonna bob over to your blog to check out that post you mentioned :)
@Colin
Cheers, sounds good.
-
My experience by using Repository pattern is that we can’t create a Generic one, the reason is that most of them aren’t unique and changed based on the context they should be used. Some basic methods can be reused by making it as a Base class. Remember that a method is their to communicate and there is no generic way to create a interface for all Repositories because of they will be changed based on which context they are used. Some Repositories may need specific methods which is unique for a particular Repository.
The IQueryable will lead to deferred execution, so you will never know when the execution will take place (You maybe will, but not when other uses your code). Be pragmatic, and writes code that others can’t use in a wrong way, and returning IQueryable can lead to problems so have that in mind. There is a reason why we want to use separation of concerns and Repository has the task to handle all the execution based on queries. If we put execution in one place, we always know where the execution will take place. Have also in mind that IQueryable is only a query object, so what you do is to pass a query between layers and somewhere during the way up to the top layer, the execution will take place, but you will not have control when it take place, and only that it one reason why I don’t like returning IQueryable.
One thing more, avoids returning List<t> or IList<t> from public members if you shouldn’t use at least 80-90% of the methods. You can read more about it here:
http://weblogs.asp.net/fredriknormen/archive/2008/11/02/don-t-return-list-lt-t-gt-from-a-public-member.aspx