It might be because you feel the pressure of temptation, or it might be because of something more intense, like you are being haunted. But in either situation the same answer applies.
You don't keep the devil away - Jesus does.
The reason the Son of God appeared was to destroy the works of the devil (1 John 3:8). We are helpless against the Evil One. We are powerless. He will always overcome us if we try to defeat him on our own. But that is why Christ comes.
Christ comes in His Word. A Word He invites you to speak. His Holy Name He invites you to speal. His holy blood He invites you to invoke. Spit God's Word at the devil. Shout if you have to. Say it outloud and believe every syllable. Christ comes in His Word.
Christ comes in His holy washing. If you are having trouble with evil thoughts, desires or temptations, dive in the cool waters of Baptism. Remember the promise God made to you there and say it outloud. The fiery darts of the Evil One must be extinguished by the wet promises of God in Baptism. And if you are not baptized - GET BAPTIZED! God wants to protect you with His holy washing, because Christ comes in His holy washing.
This may be a radical change of thinking for you. You do not take on evil by yourself. You don't even take on evil side by side with a teammate. Jesus takes on evil in your place. He takes it down by dying with it, crucifying it in His own body, ending its power against you.
And Jesus brings His protection - all the benefits of His death and resurrection - through His Word and Sacrament. So when the devil comes around, speak God's Word, remember His promise made in baptism, trust all that He has said, for the devil cannot endure God's Word.