I don't think you're being a baby at all. If I had an issue with a person and got assigned to write them a fic, I would be hard pressed to find inspiration to write the thing, no matter if it's the pairing I write with the most ease or not! The issues (or non-issues) you have with the person, IMO, could really cloud your work and likely you may not be happy with the end result.